Skip to content

Commit d9931c8

Browse files
mathomp4mathomp4
authored
Add NAS as tier1 site (#1708)
This PR adds NAS as a new site for spack-stack. These files were used to build spack-stack 1.9.0. The two stacks I was able to build are [email protected] with MPT and [email protected] with Open MPI. Note: For [email protected] I just could not build hdf-eos2 so I had to alter the met variants to skip that.
1 parent f6bdd79 commit d9931c8

File tree

10 files changed

+300
-1
lines changed

10 files changed

+300
-1
lines changed

‎configs/common/packages.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,8 @@ packages:
171171
mapl:
172172
require: '@2.53 ~shared ~f2py'
173173
variants: '+pflogger'
174+
# Note: Please check the sites/tier1/nas/packages_gcc.yaml if
175+
# this is changed, as it has a custom variant combination
174176
met:
175177
require:
176178
- '@12.0.1'

‎configs/sites/tier1/nas/README.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# How to build spack-stack at NAS
2+
3+
## Generic
4+
5+
```
6+
git clone --recursive https://github.com/JCSDA/spack-stack.git -b release/1.9.0 spack-stack-1.9.1
7+
```
8+
9+
## oneapi
10+
11+
```
12+
cd spack-stack-1.9.1
13+
. setup.sh
14+
spack stack create env --name ue-oneapi-2024.2.0 --template unified-dev --site nas --compiler oneapi
15+
cd envs/ue-oneapi-2024.2.0
16+
spack env activate .
17+
spack concretize 2>&1 | tee log.concretize
18+
spack install --verbose --fail-fast --show-log-on-error --no-check-signature 2>&1 | tee log.install
19+
```
20+
21+
NOTE: You might need to run the `spack install` command multiple times because sometimes
22+
it just fails. But then you run it more and more and it will eventually succeed.
23+
24+
```
25+
spack module tcl refresh -y
26+
spack stack setup-meta-modules
27+
spack env deactivate
28+
```
29+
30+
## gcc
31+
32+
```
33+
cd spack-stack-1.9.1
34+
. setup.sh
35+
spack stack create env --name ue-gcc-12.3.0 --template unified-dev --site nas --compiler gcc
36+
cd envs/ue-gcc-12.3.0
37+
spack env activate .
38+
spack concretize 2>&1 | tee log.concretize
39+
spack install --verbose --fail-fast --show-log-on-error --no-check-signature 2>&1 | tee log.install
40+
```
41+
42+
NOTE: You might need to run the `spack install` command multiple times because sometimes
43+
it just fails. But then you run it more and more and it will eventually succeed.
44+
45+
```
46+
spack module tcl refresh -y
47+
spack stack setup-meta-modules
48+
spack env deactivate
49+
```

‎configs/sites/tier1/nas/compilers.yaml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
compilers:
2+
- compiler:
3+
spec: oneapi@=2024.2.0
4+
paths:
5+
cc: /nobackup/gmao_SIteam/intel/oneapi/compiler/2024.2/bin/icx
6+
cxx: /nobackup/gmao_SIteam/intel/oneapi/compiler/2024.2/bin/icpx
7+
f77: /nobackup/gmao_SIteam/intel/oneapi/compiler/2024.2/bin/ifort
8+
fc: /nobackup/gmao_SIteam/intel/oneapi/compiler/2024.2/bin/ifort
9+
flags: {}
10+
operating_system: rhel8
11+
target: x86_64
12+
modules:
13+
- comp-intel/2024.2.0-ifort
14+
environment:
15+
prepend_path:
16+
PATH: '/nobackup/gmao_SIteam/gcc/gcc-12.3.0-TOSS4/bin'
17+
CPATH: '/nobackup/gmao_SIteam/gcc/gcc-12.3.0-TOSS4/include'
18+
LD_LIBRARY_PATH: '/nobackup/gmao_SIteam/intel/oneapi/compiler/2024.2/lib/:/nobackup/gmao_SIteam/gcc/gcc-12.3.0-TOSS4/lib64'
19+
set:
20+
MPI_COLL_REPRODUCIBLE: 'yes'
21+
MPI_MEMMAP_OFF: 'yes'
22+
MPI_XPMEM_ENABLED: 'yes'
23+
MPI_LAUNCH_TIMEOUT: '40'
24+
MPI_COMM_MAX: '1024'
25+
MPI_GROUP_MAX: '1024'
26+
MPI_BUFS_PER_PROC: '256'
27+
unset:
28+
- MPI_NUM_MEMORY_REGIONS
29+
- SUPPRESS_XPMEM_TRIM_THRESH
30+
- PMI_RANK
31+
extra_rpaths: []
32+
- compiler:
33+
spec: gcc@=12.3.0
34+
paths:
35+
cc: /nobackup/gmao_SIteam/gcc/gcc-12.3.0-TOSS4/bin/gcc
36+
cxx: /nobackup/gmao_SIteam/gcc/gcc-12.3.0-TOSS4/bin/g++
37+
f77: /nobackup/gmao_SIteam/gcc/gcc-12.3.0-TOSS4/bin/gfortran
38+
fc: /nobackup/gmao_SIteam/gcc/gcc-12.3.0-TOSS4/bin/gfortran
39+
flags: {}
40+
operating_system: rhel8
41+
target: x86_64
42+
modules:
43+
- comp-gcc/12.3.0-TOSS4
44+
environment:
45+
set:
46+
OMPI_MCA_mpi_preconnect_all: 1
47+
OMPI_MCA_coll_tuned_bcast_algorithm: 7
48+
OMPI_MCA_coll_tuned_scatter_algorithm: 2
49+
OMPI_MCA_coll_tuned_reduce_scatter_algorithm: 3
50+
OMPI_MCA_coll_tuned_allreduce_algorithm: 3
51+
OMPI_MCA_coll_tuned_allgather_algorithm: 4
52+
OMPI_MCA_coll_tuned_allgatherv_algorithm: 3
53+
OMPI_MCA_coll_tuned_gather_algorithm: 1
54+
OMPI_MCA_coll_tuned_barrier_algorithm: 0
55+
OMPI_MCA_coll_tuned_use_dynamic_rules: 1
56+
OMPI_MCA_sharedfp: '^lockedfile,individual'
57+
extra_rpaths: []

‎configs/sites/tier1/nas/config.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
config:
2+
build_jobs: 6

‎configs/sites/tier1/nas/mirrors.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
mirrors:
2+
local-source:
3+
fetch:
4+
url: file:///nobackup/gmao_SIteam/spack-stack/source-cache
5+
access_pair:
6+
- null
7+
- null
8+
access_token: null
9+
profile: null
10+
endpoint_url: null
11+
push:
12+
url: file:///nobackup/gmao_SIteam/spack-stack/source-cache
13+
access_pair:
14+
- null
15+
- null
16+
access_token: null
17+
profile: null
18+
endpoint_url: null

‎configs/sites/tier1/nas/modules.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
modules:
2+
default:
3+
enable::
4+
- tcl
5+
tcl:
6+
include:
7+
# List of packages for which we need modules that are blacklisted by default
8+
- python

‎configs/sites/tier1/nas/packages.yaml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
packages:
2+
all:
3+
target: [x86_64_v3]
4+
### Modification of common packages
5+
# Problems building shared hdf-eos2 with Intel, not needed
6+
hdf-eos2:
7+
variants: ~shared
8+
met:
9+
variants: +python +grib2 +graphics +lidar2nc +modis
10+
### All other external packages listed alphabetically
11+
automake:
12+
externals:
13+
- spec: [email protected]
14+
prefix: /usr
15+
binutils:
16+
externals:
17+
- spec: [email protected]~gold~headers
18+
prefix: /usr
19+
bison:
20+
externals:
21+
- spec: [email protected]
22+
prefix: /usr
23+
coreutils:
24+
externals:
25+
- spec: [email protected]
26+
prefix: /usr
27+
diffutils:
28+
externals:
29+
- spec: [email protected]
30+
prefix: /usr
31+
findutils:
32+
externals:
33+
- spec: [email protected]
34+
prefix: /usr
35+
flex:
36+
externals:
37+
- spec: [email protected]+lex
38+
prefix: /usr
39+
gawk:
40+
externals:
41+
- spec: [email protected]
42+
prefix: /usr
43+
git:
44+
externals:
45+
- spec: [email protected]~tcltk
46+
prefix: /usr
47+
gmake:
48+
externals:
49+
- spec: [email protected]
50+
prefix: /usr
51+
grep:
52+
externals:
53+
- spec: [email protected]
54+
prefix: /usr
55+
groff:
56+
externals:
57+
- spec: [email protected]
58+
prefix: /usr
59+
libtool:
60+
externals:
61+
- spec: [email protected]
62+
prefix: /usr
63+
m4:
64+
externals:
65+
- spec: [email protected]
66+
prefix: /usr
67+
perl:
68+
externals:
69+
- spec: [email protected]~cpanm+opcode+open+shared+threads
70+
prefix: /usr
71+
pkgconf:
72+
externals:
73+
- spec: [email protected]
74+
prefix: /usr
75+
sed:
76+
externals:
77+
- spec: [email protected]
78+
prefix: /usr
79+
subversion:
80+
externals:
81+
- spec: [email protected]
82+
prefix: /usr
83+
tar:
84+
externals:
85+
- spec: [email protected]
86+
prefix: /usr
87+
wget:
88+
externals:
89+
- spec: [email protected]
90+
prefix: /usr
91+
zlib:
92+
externals:
93+
- spec: [email protected]
94+
prefix: /usr

‎configs/sites/tier1/nas/packages_gcc.yaml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
packages:
2+
all:
3+
compiler:: [[email protected]]
4+
providers:
5+
mpi:: [[email protected]]
6+
mpi:
7+
buildable: False
8+
openmpi:
9+
externals:
10+
- spec: [email protected]%gcc@=12.3.0~cuda~cxx~cxx_exceptions~java~memchecker+pmi~static~wrapper-rpath+lustre
11+
fabrics=ucx schedulers=tm
12+
prefix: /nobackup/gmao_SIteam/MPI/openmpi/4.1.8/gcc-12.3.0
13+
environment:
14+
set:
15+
OPAL_PREFIX: /nobackup/gmao_SIteam/MPI/openmpi/4.1.8/gcc-12.3.0
16+
UCX_MAX_EAGER_LANES: 1
17+
UCX_MAX_RNDV_LANES: 1
18+
OMPI_MCA_pml: ucx
19+
OMPI_MCA_btl: ^openib
20+
OMPI_MCA_oob_tcp_listen_mode: listen_thread
21+
OMPI_MCA_oob_tcp_if_include: ib0,ib1
22+
OMPI_MCA_btl_tcp_if_include: ib0,ib1
23+
OMPI_MCA_opal_warn_on_missing_libcuda: 0
24+
UCX_IB_SUBNET_PREFIX: fec0:0000:0000:0008
25+
OMPI_MCA_shmem_mmap_enable_nfs_warning: 0
26+
OPENMPI: /nobackup/gmao_SIteam/MPI/openmpi/4.1.8/gcc-12.3.0
27+
prepend_path:
28+
PATH: /nobackup/gmao_SIteam/MPI/openmpi/4.1.8/gcc-12.3.0/bin
29+
LIBRARY_PATH: /nobackup/gmao_SIteam/MPI/openmpi/4.1.8/gcc-12.3.0/lib
30+
LD_LIBRARY_PATH: /nobackup/gmao_SIteam/MPI/openmpi/4.1.8/gcc-12.3.0/lib
31+
CPATH: /nobackup/gmao_SIteam/MPI/openmpi/4.1.8/gcc-12.3.0/include
32+
FPATH: /nobackup/gmao_SIteam/MPI/openmpi/4.1.8/gcc-12.3.0/include
33+
MANPATH: /nobackup/gmao_SIteam/MPI/openmpi/4.1.8/gcc-12.3.0/share/man
34+
# MAT: I cannot figure out how to get hdf-eos2 to build with GCC on NAS
35+
# When I try, it dies with an error trying to *patch* the code!
36+
# This turns off the variants of met that need it
37+
met:
38+
variants::
39+
- '+python +grib2 +graphics'

‎configs/sites/tier1/nas/packages_oneapi.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
packages:
2+
all:
3+
compiler:: [[email protected],[email protected]]
4+
providers:
5+
mpi:: [[email protected]]
6+
mpi:
7+
buildable: False
8+
mpt:
9+
externals:
10+
- spec: [email protected]%oneapi@=2024.2.0
11+
prefix: /nasa/hpe/mpt/2.30_rhel810
12+
intel-oneapi-mkl:
13+
externals:
14+
- spec: [email protected]%[email protected]
15+
prefix: /nobackup/gmao_SIteam/intel/oneapi

‎doc/source/PreConfiguredSites.rst‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,9 @@ Pre-configured sites (tier 1)
5050
| MSU +-----------------------+--------------------+--------------------------------------------------------+-----------------+
5151
| | Orion | oneAPI | ``/apps/contrib/spack-stack/`` | EPIC / JCSDA |
5252
+---------------------+-----------------------+--------------------+--------------------------------------------------------+-----------------+
53-
| NASA | Discover SCU17 | GCC, Intel | ``/gpfsm/dswdev/jcsda/spack-stack/scu17/`` | JCSDA |
53+
| | Discover SCU17 | GCC, Intel | ``/gpfsm/dswdev/jcsda/spack-stack/scu17/`` | JCSDA |
54+
| NASA +-----------------------+--------------------+--------------------------------------------------------+-----------------+
55+
| | NAS | GCC, Intel | ``/nobackup/gmao_SIteam/spack-stack/`` | GMAO |
5456
+---------------------+-----------------------+--------------------+--------------------------------------------------------+-----------------+
5557
| NCAR-Wyoming + Derecho | GCC, Intel | ``/glade/work/epicufsrt/contrib/spack-stack/derecho/`` | EPIC / JCSDA |
5658
+---------------------+-----------------------+--------------------+--------------------------------------------------------+-----------------+
@@ -133,6 +135,19 @@ The following is required for building new spack environments with any supported
133135
module use /discover/swdev/gmao_SIteam/modulefiles-SLES15
134136
module use /discover/swdev/jcsda/spack-stack/scu17/modulefiles
135137
138+
.. _Preconfigured_Sites_NAS:
139+
140+
------------------------------
141+
NASA NAS
142+
------------------------------
143+
144+
The following is required for building new spack environments with any supported compiler on this platform.
145+
146+
.. code-block:: console
147+
148+
module purge
149+
module use /nobackup/gmao_SIteam/modulefiles
150+
136151
.. _Preconfigured_Sites_Narwhal:
137152

138153
------------------------------

0 commit comments

Comments
 (0)