Skip to main content
Log in

Modeling the propagation of elastic waves using spectral elements on a cluster of 192 GPUs

  • Special Issue Paper
  • Published:
Computer Science - Research and Development

Abstract

We implement a high-order finite-element application, which performs the numerical simulation of seismic wave propagation resulting for instance from earthquakes at the scale of a continent or from active seismic acquisition experiments in the oil industry, on a large GPU-enhanced cluster. Mesh coloring enables an efficient accumulation of degrees of freedom in the assembly process over an unstructured mesh. We use non-blocking MPI and show that computations and communications over the network and between the CPUs and the GPUs are almost fully overlapped. The GPU solver scales excellently up to 192 GPUs and achieves significant speedup over a carefully tuned equivalent CPU code.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdelkhalek R, Calandra H, Coulaud O, Roman J, Latu G (2009) Fast seismic modeling and reverse time migration on a GPU cluster. In: Smari WW, McIntire JP (eds) High performance computing & simulation 2009, pp 36–44. http://hal.inria.fr/docs/00/40/39/33/PDF/hpcs.pdf

  2. Carrington L, Komatitsch D, Laurenzano M, Tikir M, Michéa D, Le Goff N, Snavely A, Tromp J (2008) High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62k processors. In: SC ’08: proceedings of the 2008 ACM/IEEE conference on supercomputing, article #60, Gordon Bell Prize finalist article

  3. Chaljub E, Komatitsch D, Vilotte JP, Capdeville Y, Valette B, Festa G (2007) Spectral element analysis in seismology. In: Wu RS, Maupin V (eds) Advances in wave propagation in heterogeneous media. Advances in geophysics, vol 48. Elsevier/Academic Press, Amsterdam/San Diego, pp 365–419

    Chapter  Google Scholar 

  4. Che S, Boyer M, Meng J, Tarjan D, Sheaffer JW, Skadron K (2008) A performance study of general-purpose applications on graphics processors using CUDA. J Parall Distrib Comput 68(10):1370–1380. doi:10.1016/j.jpdc.2008.05.014

    Article  Google Scholar 

  5. Corporation NVIDIA (2009) NVIDIA CUDA programming guide version 2.3. Santa Clara, California, USA, URL http://www.nvidia.com/cuda

  6. De Basabe JD, Sen MK (2007) Grid dispersion and stability criteria of some common finite-element methods for acoustic and elastic wave equations. Geophysics 72(6):T81–T95. doi:10.1190/1.2785046

    Article  Google Scholar 

  7. Dziewoński AM, Anderson DL (1981) Preliminary reference Earth model. Phys Earth Planet Inter 25(4):297–356. doi:10.1016/0031-9201(81)90046-7

    Article  Google Scholar 

  8. Fan Z, Qiu F, Kaufman AE, Yoakum-Stover S (2004) GPU cluster for high performance computing. In: SC ’04: proceedings of the 2004 ACM/IEEE conference on supercomputing, p 47. doi:10.1109/SC.2004.26

  9. Fatahalian K, Houston M (2008) A closer look at GPUs. Commun ACM 51(10):50–57. doi:10.1145/1400181.1400197

    Article  Google Scholar 

  10. Garland M, Grand SL, Nickolls J, Anderson JA, Hardwick J, Morton S, Phillips EH, Zhang Y, Volkov V (2008) Parallel computing experiences with CUDA. IEEE Micro 28(4):13–27. doi:10.1109/MM.2008.57

    Article  Google Scholar 

  11. Göddeke D, Strzodka R, Mohd-Yusof J, McCormick P, Buijssen SHM, Grajewski M, Turek S (2007) Exploring weak scalability for FEM calculations on a GPU-enhanced cluster. Parall Comput 33(10–11):685–699. doi:10.1016/j.parco.2007.09.002

    Google Scholar 

  12. Hughes TJR (1987) The finite element method, linear static and dynamic finite element analysis. Prentice Hall International, Englewood Cliffs

    MATH  Google Scholar 

  13. Kindratenko VV, Enos JJ, Shi G, Showerman MT, Arnold GW, Stone JE, Phillips JC, Hwu W (2009) GPU clusters for high-performance computing. In: Proceedings on the IEEE cluster’2009 workshop on parallel programming on accelerator clusters (PPAC’09), pp 1–8

  14. Komatitsch D, Coutel F, Mora P (1996) Tensorial formulation of the wave equation for modelling curved interfaces. Geophys J Int 127(1):156–168. doi:10.1111/j.1365-246X.1996.tb01541.x

    Article  Google Scholar 

  15. Komatitsch D, Tsuboi S, Ji C, Tromp J (2003) A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator. In: SC ’03: proceedings of the 2003 ACM/IEEE conference on supercomputing, pp 4–11, Gordon Bell Prize winner article. doi:10.1109/SC.2003.10023

  16. Komatitsch D, Labarta J, Michéa D (2008) A simulation of seismic wave propagation at high resolution in the inner core of the Earth on 2166 processors of MareNostrum. In: High performance computing for computational science—VECPAR 2008. Lecture notes in computer science, vol 5336. Springer, Berlin, pp 364–377. doi:10.1007/978-3-540-92859-1_33

    Chapter  Google Scholar 

  17. Komatitsch D, Michéa D, Erlebacher G (2009) Porting a high-order finite-element earthquake modeling application to NVIDIA graphics cards using CUDA. J Parall Distrib Comput 69(5):451–460. doi:10.1016/j.jpdc.2009.01.006

    Article  Google Scholar 

  18. Liu Q, Polet J, Komatitsch D, Tromp J (2004) Spectral-element moment tensor inversions for earthquakes in Southern California. Bull Seismol Soc Am 94(5):1748–1761. doi:10.1785/012004038

    Article  Google Scholar 

  19. Michéa D, Komatitsch D (2010) Accelerating a 3D finite-difference wave propagation code using graphics cards. Geophys J Int (in press)

  20. Micikevicius P (2009) 3D finite-difference computation on GPUs using CUDA. In: GPGPU-2: proceedings of the 2nd workshop on general purpose processing on graphics processing units, pp 79–84. doi:10.1145/1513895.1513905

  21. Nissen-Meyer T, Fournier A, Dahlen FA (2008) A 2-D spectral-element method for computing spherical-earth seismograms—II. Waves in solid-fluid media. Geophys J Int 174(3):873–888. doi:10.1111/j.1365-246X.2008.03813.x

    Article  Google Scholar 

  22. Owens JD, Houston M, Luebke DP, Green S, Stone JE, Phillips JC (2008) GPU computing. Proc IEEE 96(5):879–899. doi:10.1109/JPROC.2008.917757

    Article  Google Scholar 

  23. Phillips JC, Stone JE, Schulten K (2008) Adapting a message-driven parallel application to GPU-accelerated clusters. In: SC ’08: proceedings of the 2008 ACM/IEEE conference on supercomputing, article no 8. doi:10.1145/1413370.1413379

  24. Phillips EH, Zhang Y, Davis RL, Owens JD (2009) Rapid aerodynamic performance prediction on a cluster of graphics processing units. In: Proceedings of the 47th AIAA aerospace sciences meeting, aIAA 2009-565

  25. Seriani G, Priolo E (1994) A spectral element method for acoustic wave simulation in heterogeneous media. Finite Elem Anal Des 16(3–4):337–348. doi:10.1016/0168-874X(94)90076-0

    Article  MATH  MathSciNet  Google Scholar 

  26. Tromp J, Komatitsch D, Liu Q (2008) Spectral-element and adjoint methods in seismology. Commun Comput Phys 3(1):1–32

    MATH  Google Scholar 

  27. van Wijk K, Komatitsch D, Scales JA, Tromp J (2004) Analysis of strong scattering at the micro-scale. J Acoust Soc Am 115(3):1006–1011. doi:10.1121/1.1647480

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dimitri Komatitsch.

Additional information

This research was funded in part by French ANR grant NUMASIS ANR-05-CIGC-002, by French CNRS, INRIA and IUF, by German Deutsche Forschungsgemeinschaft projects TU102/22-1 and TU102/22-2, and by German Bundesministerium für Bildung und Forschung in the SKALB project 01IH08003D of call ‘HPC Software für skalierbare Parallelrechner’.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Komatitsch, D., Göddeke, D., Erlebacher, G. et al. Modeling the propagation of elastic waves using spectral elements on a cluster of 192 GPUs. Comput Sci Res Dev 25, 75–82 (2010). https://doi.org/10.1007/s00450-010-0109-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00450-010-0109-1

Keywords

Navigation