diff --git a/Docs/sphinx_documentation/source/BuildingAMReX.rst b/Docs/sphinx_documentation/source/BuildingAMReX.rst index 064d11e740a..1ea35b7198a 100644 --- a/Docs/sphinx_documentation/source/BuildingAMReX.rst +++ b/Docs/sphinx_documentation/source/BuildingAMReX.rst @@ -815,3 +815,114 @@ use the install commands, .. code-block:: bash spack install amrex@develop dimensions=2 +cuda cuda_arch=60 + +.. _sec:build:amrex-kokkos: + +AMReX--Kokkos +============= +There is an `example in amrex-devtests`_ of how one can build `Kokkos`_ and `AMReX`_. This uses ``cmake`` to first compile +Kokkos and AMReX, and then the example is built. This section describes the build procedure on `Perlmutter`_, though this can be used as +a template for other machines as well. + +Load the following modules, and specify ``MPI_INCLUDE_PATH`` in ``~/.bash_profile``. The ``cmake`` version has to be ``3.24.3``. + +.. code-block:: bash + + module load cray-mpich + module load PrgEnv-gnu + module load cudatoolkit/12.2 + module load cmake/3.24.3 + + export MPI_INCLUDE_PATH=/opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3/include + +Make sure to do ``source ~/.bash_profile``. + +Kokkos installation on Perlmutter +--------------------------------- +To install `Kokkos`_, execute the following commands. In the ``cmake`` command, specify the path where the Kokkos installation should reside +``-DCMAKE_INSTALL_PREFIX=``. The full path to the ``kokkos`` directory has to be specified in +``-DCMAKE_CXX_COMPILER=/bin/nvcc_wrapper``. + +.. code-block:: bash + + git clone https://github.com/kokkos/kokkos.git + cd kokkos + mkdir build + cd build + cmake .. -DCMAKE_INSTALL_PREFIX= -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 -DKokkos_ENABLE_CUDA=ON -DCMAKE_CXX_COMPILER=/bin/nvcc_wrapper -DKokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE=ON -DKokkos_ARCH_PASCAL60=ON + make -j8 + make install + +AMReX installation on Perlmutter +--------------------------------- +.. note:: + + After cloning the repository in the first step below, add the following lines to ``amrex/CMakeLists.txt`` for MPI installation. + + .. code-block:: bash + + # Find MPI + find_package(MPI REQUIRED) + + # Include MPI headers + include_directories(${MPI_INCLUDE_PATH}) + +To install `AMReX`_, execute the following commands. In the ``cmake`` command, specify the path where the installation AMReX installation should reside +``-DCMAKE_INSTALL_PREFIX=``. + +.. code-block:: bash + + git clone https://github.com/AMReX-Codes/amrex.git + cd amrex + mkdir build + cd build + cmake .. -DCMAKE_INSTALL_PREFIX= -DAMReX_GPU_BACKEND=CUDA -DAMReX_CUDA_ARCH=60 -DAMReX_MPI=OFF -DCMAKE_PREFIX_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.9/math_libs/12.2/lib64 -DAMReX_MPI=ON -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DMPI_INCLUDE_PATH=/opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3/include + make -j8 + make install + +Compiling the AMReX-Kokkos example on Perlmutter +---------------------------------------------------- + +.. note:: + + After cloning the repository in the first step below, add the following lines to ``amrex-devtests/kokkos/CMakeLists.txt`` for MPI installation. + + .. code-block:: bash + + # Find MPI + find_package(MPI REQUIRED) + + # Include MPI headers + include_directories(${MPI_INCLUDE_PATH}) + +To compile the AMReX-Kokkos example, execute the following commands. In the ``cmake`` command, specify the full path to the amrex installation +directory ``-DAMReX_ROOT=``, and the Kokkos installation directory ``-DKokkos_ROOT=``. + +.. code-block:: bash + + git clone https://github.com/WeiqunZhang/amrex-devtests.git + cd amrex-devtests/kokkos + mkdir build + cd build + cmake .. -DENABLE_CUDA=ON -DAMReX_ROOT= -DKokkos_ROOT= -DCMAKE_CUDA_ARCHITECTURES=60 -DMPI_INCLUDE_PATH=/opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3/include + make -j8 + +Running the example on Perlmutter +------------------------------------- +Get an interactive node on Perlmutter + +.. code-block:: bash + + salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 4 --account= + +Run using + +.. code-block:: bash + + srun -n 4 --gpus=4 + +.. _`example in amrex-devtests`: https://github.com/WeiqunZhang/amrex-devtests/tree/main/kokkos +.. _`Kokkos`: https://github.com/kokkos/kokkos +.. _`AMReX`: https://github.com/AMReX-Codes/amrex +.. _`Perlmutter`: https://docs.nersc.gov/systems/perlmutter/architecture/ +