Building with GPU Support¶
FrontISTR can run on GPUs using the OpenACC functionality of the NVIDIA HPC SDK. This page describes how to build FrontISTR with GPU support enabled and how to run it.
For the prerequisite libraries, see Required and Optional Dependencies.
Build Procedure¶
Use the NVIDIA HPC SDK compilers (nvc, nvc++, and nvfortran). In an HPC cluster environment, the NVIDIA HPC SDK must be loaded before building according to the environment manual (for example, module load nvhpc).
Optional libraries such as METIS, MUMPS, and MKL do not support GPUs, so they are disabled in the command examples below (for example, WITH_METIS=OFF).
As described in Building with CMake, when the NVIDIA HPC SDK (nvc, nvc++, nvfortran) is specified for the compilers, -gpu=ccnative,mem:managed is set in CMakeLists.txt. ccnative is an option that automatically detects the GPU in the build environment and selects the optimal target, so no manual changes are required for each environment. mem: is an option that specifies how GPU memory is handled; it is required because the FrontISTR GPU implementation assumes Unified Memory. mem:managed is the default setting for general environments in which the CPU and GPU have separate memory.
Note: On architectures such as Miyabi (GH200), where the CPU and GPU are integrated via NVLink, instead of
mem:managed,mem:unifiedis used; change the setting according to the environment in use (-gpu=ccnative,mem:unified).
OpenACC + OpenMP (without MPI)¶
mkdir build
cd build
CC=nvc CXX=nvc++ FC=nvfortran cmake \
-DWITH_MPI=OFF -DWITH_OPENMP=ON \
-DWITH_METIS=OFF -DWITH_MUMPS=OFF \
-DWITH_MKL=OFF -DWITH_LAPACK=OFF \
-DWITH_ML=OFF -DWITH_REFINER=OFF -DWITH_REVOCAP=OFF \
..
make -j
OpenACC + OpenMP + MPI¶
When using MPI, the MPI wrapper compilers (such as mpicc) must be configured to invoke the NVIDIA HPC SDK compilers (such as nvc) internally. In many HPC cluster environments, loading the NVIDIA HPC SDK and MPI libraries with commands such as module load automatically provides this configuration. However, recommended versions and loading procedures vary by environment, so refer to the user manual or other documentation for each environment for details.
mkdir build
cd build
CC=mpicc CXX=mpic++ FC=mpif90 cmake \
-DWITH_MPI=ON -DWITH_OPENMP=ON \
-DWITH_METIS=OFF -DWITH_MUMPS=OFF \
-DWITH_MKL=OFF -DWITH_LAPACK=OFF \
-DWITH_ML=OFF -DWITH_REFINER=OFF -DWITH_REVOCAP=OFF \
..
make -j
Execution¶
The execution procedure for a GPU build is basically the same as for a CPU build. For general execution procedures, see the Execution Guide.
Profiling¶
Profiling data can be collected using NVIDIA Nsight Systems (nsys).