Open Accelerated Computing Summit 2023

Lucas Gasparino Ferreira da Silva, Barcelona Supercomputing Center

SOD2D is a code developed at BSC-CNS for performing large eddy simulation (LES) and direct numerical simulation (DNS) of compressible turbulent flows in realistic complex cases, with a particular focus on the aviation industry. It employs a high-order Spectral continuous Galerkin, SEM for short, discretization in conjunction with the entropy viscosity method adapted to SEM to achieve high accuracy and robustness at a reasonable computational cost. The aim of this project is to try and bridge the gap between industry needs and highly accurate flow simulations, which requires the use of high-fidelity models and large meshes. This in turn requires the use of HPC systems, which are increasingly relying on GPUs to achieve the required performance. 

In this talk, we will present how we used OpenACC to allow our scale-resolving CFD code to run efficiently on multiple GPUs. Using OpenACC to port most of the code to GPUs (in particular, all of the compute algorithm), we were able to achieve excellent performance on NVIDIA GPUs, including the latest A100 and H100 architectures. As a preview of some of the most striking results, we were able to assess that a single H100 GPU can run a heavily refined mesh (~ 48M nodes) composed of 3rd order hexahedra at a cost of about 370ms per time-step, a feat that would require several nodes of the MareNostrum4 supercomputer at BSC-CNS. We will also present scalability data demonstrating how well the code performs when running on multiple GPUs, both when communicating data using MPI and NVIDIA Collective Communications Library (NCCL) in tandem with OpenACC directives.

Igor Sfiligoi, University of California, San Diego

NVIDIA has been the main provider of GPU hardware in HPC systems for over a decade. Many applications that benefit from GPUs have thus been developed and optimized for the NVIDIA software stack. Recent exascale HPC systems are, however, introducing GPUs from other vendors, such as the AMD GPU-based Oak Ridge Leadership Computing Facility (OLCF) Frontier system recently becoming available. AMD GPUs cannot be directly accessed using the NVIDIA software stack and require a porting effort by the application developers. 

This talk provides an overview of our experience porting and optimizing the CGYRO code, a widely-used fusion simulation tool based on Fortran with OpenACC-based GPU acceleration. While the porting from the NVIDIA compilers was relatively straightforward using the CRAY compilers on the AMD systems, the performance optimization required more fine-tuning. In the optimization effort, we uncovered code sections that had performed well on NVIDIA GPUs but were unexpectedly slow on AMD GPUs. After AMD-targeted code optimizations, performance on AMD GPUs has increased to meet our expectations. Modest speed improvements were also seen on NVIDIA GPUs, which was an unexpected benefit of this exercise.

Victor Yu, Argonne National Laboratory

Many-body perturbation theory (MBPT) is a powerful method for simulating electronic excitations in molecules and materials. In this talk, we present a massively parallel, GPU-accelerated implementation of MBPT in the WEST code (www.west-code.org). Outstanding performance and scalability are achieved by employing a hierarchical parallelization strategy, nonblocking MPI communications, and mixed precision in selected portions of the code. The capability of the GPU version of WEST is demonstrated by large-scale MBPT calculations using up to 25,920 GPUs. Finally, we delve into our experience of switching our GPU programming model from CUDA to OpenACC, which is enabling us to attain enhanced performance portability. 

Joel Denny, OpenACC Co-Chair of the Technical Committee, Oak Ridge National Laboratory

Clacc has developed OpenACC compiler, runtime, and profiling support for C and C++ by extending Clang and LLVM under the Exascale Computing Project (ECP).  OpenACC support in Clang and LLVM can facilitate the programming of GPUs and other accelerators in HPC applications and provide a popular compiler platform on which to perform research and development for related optimizations and tools for heterogeneous computing architectures.  A key Clacc design decision is to translate OpenACC to OpenMP to leverage the OpenMP offloading support that is actively being developed for Clang and LLVM.  A benefit of this design is support for two compilation modes: a traditional compilation mode that translates OpenACC source to an executable, and a source-to-source mode that translates OpenACC source to OpenMP source.  Clacc is hosted publicly on GitHub as part of the LLVM Department of Energy (DOE) Fork maintained at Oak Ridge National Laboratory (ORNL) (https://github.com/llvm-doe-org/llvm-project/wiki).

This talk presents the latest developments in the Clacc project as well as future plans in light of the end of ECP later this year.  We will cover topics including recent Clacc support for OpenACC in C++, support for KokkACC (a new OpenACC backend for Kokkos which won the best paper award at WACCPD 2022), and a general summary of Clacc's current OpenACC feature support.  We will also invite the community to give feedback on their interest in seeing OpenACC support in the LLVM ecosystem going forward.

Pedro Costa, Delft University of Technology

Direct numerical simulations of the Navier-Stokes equations have greatly enhanced our understanding of turbulent fluid flows, impacting numerous environmental and industrial applications. Still, important unresolved issues remain, requiring massive computing power that just became within reach with GPU computing at scale. 

This talk focuses on the GPU porting effort of the numerical solver CaNS, a code for fast massively parallel simulations of canonical fluid flows that has gained popularity throughout the years. CaNS is written in modern Fortran and recently featured a fresh GPU porting using OpenACC and a hardware-adaptive pencil domain decomposition library. We exploited OpenACC directives for host/device data movement, loop offloading, asynchronous kernel launching, and interoperability with CUDA and external GPU libraries. More importantly, in the porting process, several interesting practices have been found where standard Fortran combined with OpenACC secures a sustainable and flexible implementation while retaining the efficiency of the numerical tool. The exchange between domain-specific and GPU computing experts was key to the success of this effort. 

We will cover the high-level implementation and performance of CaNS, including how OpenACC enabled swift interoperability with the external libraries that enabled a hardware-adaptive implementation. Additionally, we will discuss fine implementation details, highlighting simple yet impactful approaches applicable to other applications which have not been widely documented. Finally, we will demonstrate performance and show how CaNS is being used on the supercomputer Leonardo to simulate wall turbulence at unprecedented Reynolds numbers (i.e., high flow speeds).