Abstract
We implement a high-order finite-element application, which performs the numerical simulation of seismic wave propagation resulting for instance from earthquakes at the scale of a continent or from active seismic acquisition experiments in the oil industry, on a large GPU-enhanced cluster. Mesh coloring enables an efficient accumulation of degrees of freedom in the assembly process over an unstructured mesh. We use non-blocking MPI and show that computations and communications over the network and between the CPUs and the GPUs are almost fully overlapped. The GPU solver scales excellently up to 192 GPUs and achieves significant speedup over a carefully tuned equivalent CPU code.
Similar content being viewed by others
References
Abdelkhalek R, Calandra H, Coulaud O, Roman J, Latu G (2009) Fast seismic modeling and reverse time migration on a GPU cluster. In: Smari WW, McIntire JP (eds) High performance computing & simulation 2009, pp 36–44. http://hal.inria.fr/docs/00/40/39/33/PDF/hpcs.pdf
Carrington L, Komatitsch D, Laurenzano M, Tikir M, Michéa D, Le Goff N, Snavely A, Tromp J (2008) High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62k processors. In: SC ’08: proceedings of the 2008 ACM/IEEE conference on supercomputing, article #60, Gordon Bell Prize finalist article
Chaljub E, Komatitsch D, Vilotte JP, Capdeville Y, Valette B, Festa G (2007) Spectral element analysis in seismology. In: Wu RS, Maupin V (eds) Advances in wave propagation in heterogeneous media. Advances in geophysics, vol 48. Elsevier/Academic Press, Amsterdam/San Diego, pp 365–419
Che S, Boyer M, Meng J, Tarjan D, Sheaffer JW, Skadron K (2008) A performance study of general-purpose applications on graphics processors using CUDA. J Parall Distrib Comput 68(10):1370–1380. doi:10.1016/j.jpdc.2008.05.014
Corporation NVIDIA (2009) NVIDIA CUDA programming guide version 2.3. Santa Clara, California, USA, URL http://www.nvidia.com/cuda
De Basabe JD, Sen MK (2007) Grid dispersion and stability criteria of some common finite-element methods for acoustic and elastic wave equations. Geophysics 72(6):T81–T95. doi:10.1190/1.2785046
Dziewoński AM, Anderson DL (1981) Preliminary reference Earth model. Phys Earth Planet Inter 25(4):297–356. doi:10.1016/0031-9201(81)90046-7
Fan Z, Qiu F, Kaufman AE, Yoakum-Stover S (2004) GPU cluster for high performance computing. In: SC ’04: proceedings of the 2004 ACM/IEEE conference on supercomputing, p 47. doi:10.1109/SC.2004.26
Fatahalian K, Houston M (2008) A closer look at GPUs. Commun ACM 51(10):50–57. doi:10.1145/1400181.1400197
Garland M, Grand SL, Nickolls J, Anderson JA, Hardwick J, Morton S, Phillips EH, Zhang Y, Volkov V (2008) Parallel computing experiences with CUDA. IEEE Micro 28(4):13–27. doi:10.1109/MM.2008.57
Göddeke D, Strzodka R, Mohd-Yusof J, McCormick P, Buijssen SHM, Grajewski M, Turek S (2007) Exploring weak scalability for FEM calculations on a GPU-enhanced cluster. Parall Comput 33(10–11):685–699. doi:10.1016/j.parco.2007.09.002
Hughes TJR (1987) The finite element method, linear static and dynamic finite element analysis. Prentice Hall International, Englewood Cliffs
Kindratenko VV, Enos JJ, Shi G, Showerman MT, Arnold GW, Stone JE, Phillips JC, Hwu W (2009) GPU clusters for high-performance computing. In: Proceedings on the IEEE cluster’2009 workshop on parallel programming on accelerator clusters (PPAC’09), pp 1–8
Komatitsch D, Coutel F, Mora P (1996) Tensorial formulation of the wave equation for modelling curved interfaces. Geophys J Int 127(1):156–168. doi:10.1111/j.1365-246X.1996.tb01541.x
Komatitsch D, Tsuboi S, Ji C, Tromp J (2003) A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator. In: SC ’03: proceedings of the 2003 ACM/IEEE conference on supercomputing, pp 4–11, Gordon Bell Prize winner article. doi:10.1109/SC.2003.10023
Komatitsch D, Labarta J, Michéa D (2008) A simulation of seismic wave propagation at high resolution in the inner core of the Earth on 2166 processors of MareNostrum. In: High performance computing for computational science—VECPAR 2008. Lecture notes in computer science, vol 5336. Springer, Berlin, pp 364–377. doi:10.1007/978-3-540-92859-1_33
Komatitsch D, Michéa D, Erlebacher G (2009) Porting a high-order finite-element earthquake modeling application to NVIDIA graphics cards using CUDA. J Parall Distrib Comput 69(5):451–460. doi:10.1016/j.jpdc.2009.01.006
Liu Q, Polet J, Komatitsch D, Tromp J (2004) Spectral-element moment tensor inversions for earthquakes in Southern California. Bull Seismol Soc Am 94(5):1748–1761. doi:10.1785/012004038
Michéa D, Komatitsch D (2010) Accelerating a 3D finite-difference wave propagation code using graphics cards. Geophys J Int (in press)
Micikevicius P (2009) 3D finite-difference computation on GPUs using CUDA. In: GPGPU-2: proceedings of the 2nd workshop on general purpose processing on graphics processing units, pp 79–84. doi:10.1145/1513895.1513905
Nissen-Meyer T, Fournier A, Dahlen FA (2008) A 2-D spectral-element method for computing spherical-earth seismograms—II. Waves in solid-fluid media. Geophys J Int 174(3):873–888. doi:10.1111/j.1365-246X.2008.03813.x
Owens JD, Houston M, Luebke DP, Green S, Stone JE, Phillips JC (2008) GPU computing. Proc IEEE 96(5):879–899. doi:10.1109/JPROC.2008.917757
Phillips JC, Stone JE, Schulten K (2008) Adapting a message-driven parallel application to GPU-accelerated clusters. In: SC ’08: proceedings of the 2008 ACM/IEEE conference on supercomputing, article no 8. doi:10.1145/1413370.1413379
Phillips EH, Zhang Y, Davis RL, Owens JD (2009) Rapid aerodynamic performance prediction on a cluster of graphics processing units. In: Proceedings of the 47th AIAA aerospace sciences meeting, aIAA 2009-565
Seriani G, Priolo E (1994) A spectral element method for acoustic wave simulation in heterogeneous media. Finite Elem Anal Des 16(3–4):337–348. doi:10.1016/0168-874X(94)90076-0
Tromp J, Komatitsch D, Liu Q (2008) Spectral-element and adjoint methods in seismology. Commun Comput Phys 3(1):1–32
van Wijk K, Komatitsch D, Scales JA, Tromp J (2004) Analysis of strong scattering at the micro-scale. J Acoust Soc Am 115(3):1006–1011. doi:10.1121/1.1647480
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was funded in part by French ANR grant NUMASIS ANR-05-CIGC-002, by French CNRS, INRIA and IUF, by German Deutsche Forschungsgemeinschaft projects TU102/22-1 and TU102/22-2, and by German Bundesministerium für Bildung und Forschung in the SKALB project 01IH08003D of call ‘HPC Software für skalierbare Parallelrechner’.
Rights and permissions
About this article
Cite this article
Komatitsch, D., Göddeke, D., Erlebacher, G. et al. Modeling the propagation of elastic waves using spectral elements on a cluster of 192 GPUs. Comput Sci Res Dev 25, 75–82 (2010). https://doi.org/10.1007/s00450-010-0109-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00450-010-0109-1