Enabling execution of a legacy CFD mini application on accelerators using OpenMP

Ioannis Nompelis, Gabriele Jost, Alice Koniges, Christopher Daley, David Eder, Christopher Stone

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We describe the process and outcome of our efforts to port a legacy Fortran benchmark code to heterogeneous GPU-accelerated computing architectures using OpenMP. The benchmark code is one of the multi-zone NAS Parallel Benchmarks (NPB-MZ) called SP-MZ. This “mini-app” mimics the computation and data movement that is found in popular legacy and modern implicit computational fluid dynamics (CFD) solvers. Our objective was to examine how efficiently legacy Fortran codes can be ported to accelerators by leveraging OpenMP directives. We describe the development and optimization process and demonstrate the performance impact of various code modifications. We show select profiling results from the NVIDIA Visual Profiler (nvpp) to help others diagnose and overcome performance issues in their own applications. We present results for two compute systems endowed with NVIDIA V100 accelerators.

Original languageEnglish (US)
Title of host publicationHigh Performance Computing - 35th International Conference, ISC High Performance 2020, Proceedings
EditorsPonnuswamy Sadayappan, Bradford L. Chamberlain, Guido Juckeland, Hatem Ltaief
Number of pages18
ISBN (Print)9783030507428
StatePublished - 2020
Event35th International Conference on High Performance Computing, ISC High Performance 2020 - Frankfurt, Germany
Duration: Jun 22 2020Jun 25 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12151 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference35th International Conference on High Performance Computing, ISC High Performance 2020

Bibliographical note

Funding Information:
The authors would like to thank all those who helped with valuable advice during the NERSC/OLCF Oakland Hackathon, especially, Kevin Gott and Jack Deslippe from NERSC and Tom Papatheodore from OLCF for facilitation of an excellent hackathon experience, Max Katz and Angela Chen from NVIDIA who taught us a lot about profiling, and Fazlay Rabbi from NERSC who helped us with compiling and benchmarking on the Ascent and Cori systems. Thanks to Joel Bretheim from the Naval Research Lab, who contributed to the code optimizations. Special thanks go to Kelvin Li from IBM for advising on the changes of Fortran value versus by-reference arguments. I. Nompelis would like to thank Prof. G. V. Candler’s group from the Department of Aerospace Engineering and Mechanics, Univ. of Minnesota for their support. This research used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility operated under Contract No. DE-AC02-05CH11231. This research also used resources of the Oak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility supported under Contract No. DE-AC05-00OR22725. We thank OLCF and NERSC for their support during our study. This work was partially funded by NASA Contract No: 80ARC018D001. Work at the hackathon included funding from the Department of Defense High Performance Computing Modernization Program (HPCMP) under User Productivity, Technology Transfer and Training (PETTT) Contract No. GS04T09DBC0017 and for the University of Hawai‘i under Maui High Performance Computing Center (MHPCC) Contract No. N00024-19-D-6400.

Publisher Copyright:
© Springer Nature Switzerland AG 2020.


  • Accelerator
  • Fortran
  • GPU
  • Implicit CFD
  • OpenMP


Dive into the research topics of 'Enabling execution of a legacy CFD mini application on accelerators using OpenMP'. Together they form a unique fingerprint.

Cite this