Skip to content

Commit 055fed4

Browse files
committed
Merge branch 'development' into select_actual_neighbors_fix
2 parents e931b01 + 5102120 commit 055fed4

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+1093
-365
lines changed

.github/workflows/cuda.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ jobs:
2020
- uses: actions/checkout@v5
2121
- name: Dependencies
2222
run: |
23+
.github/workflows/dependencies/ubuntu_free_disk_space.sh
2324
.github/workflows/dependencies/dependencies_nvcc.sh 12.6
2425
.github/workflows/dependencies/dependencies_ccache.sh
2526
- name: Set Up Cache
@@ -82,6 +83,7 @@ jobs:
8283
- uses: actions/checkout@v5
8384
- name: Dependencies
8485
run: |
86+
.github/workflows/dependencies/ubuntu_free_disk_space.sh
8587
.github/workflows/dependencies/dependencies_llvm_cuda11_clang15.sh
8688
.github/workflows/dependencies/dependencies_ccache.sh
8789
- name: Set Up Cache
@@ -193,6 +195,7 @@ jobs:
193195
- uses: actions/checkout@v5
194196
- name: Dependencies
195197
run: |
198+
.github/workflows/dependencies/ubuntu_free_disk_space.sh
196199
.github/workflows/dependencies/dependencies_nvcc_2404.sh 13.0
197200
.github/workflows/dependencies/dependencies_ccache.sh
198201
- name: Set Up Cache

.github/workflows/gcc.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -311,7 +311,7 @@ jobs:
311311
restore-keys: |
312312
ccache-${{ github.workflow }}-${{ github.job }}-git-
313313
- name: Build & Install
314-
env: {CXXFLAGS: "-fno-operator-names -Werror -Wall -Wextra -Wpedantic -Wnull-dereference -Wfloat-conversion -Wshadow -Woverloaded-virtual -Wnon-virtual-dtor -Wlogical-op -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches -Wmissing-include-dirs -Wno-array-bounds"}
314+
env: {CXXFLAGS: "-fno-operator-names -Werror -Wall -Wextra -Wpedantic -Wnull-dereference -Wfloat-conversion -Wshadow -Woverloaded-virtual -Wnon-virtual-dtor -Wlogical-op -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches -Wmissing-include-dirs -Wno-array-bounds -Wno-maybe-uninitialized"}
315315
run: |
316316
export CCACHE_COMPRESS=1
317317
export CCACHE_COMPRESSLEVEL=10
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
name: HPSF GitLab CI
2+
3+
on:
4+
issue_comment:
5+
types: [created]
6+
7+
env:
8+
GITLAB_PROJECT_PATH: "amrex/amrex"
9+
GITLAB_PROJECT_ID: "70"
10+
11+
jobs:
12+
run-hpsf-gitlab-ci:
13+
if: |
14+
github.event.issue.pull_request &&
15+
contains(fromJson('["atmyers","asalmgren","ax3l","WeiqunZhang"]'), github.actor) &&
16+
contains(github.event.comment.body, '/run-hpsf-gitlab-ci')
17+
runs-on: ubuntu-latest
18+
19+
steps:
20+
- name: Determine PR merge ref
21+
id: setref
22+
shell: bash
23+
env:
24+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
25+
run: |
26+
PR_NUMBER=${{ github.event.issue.number }}
27+
MERGE_REF="refs/pull/$PR_NUMBER/merge"
28+
PR_TITLE=$(gh api repos/${{ github.repository }}/pulls/$PR_NUMBER --jq '.title')
29+
echo "merge_ref=$MERGE_REF" >> $GITHUB_OUTPUT
30+
echo "pr_title=$PR_TITLE" >> $GITHUB_OUTPUT
31+
echo "Using merge ref: $MERGE_REF"
32+
33+
- name: Trigger GitLab pipeline
34+
id: trigger
35+
shell: bash
36+
run: |
37+
MERGE_REF="${{ steps.setref.outputs.merge_ref }}"
38+
echo "Triggering GitLab pipeline for ref: $MERGE_REF"
39+
40+
GITHUB_PIPELINE_NAME="GitHub #${{ github.event.issue.number }}: ${{ steps.setref.outputs.pr_title }}"
41+
42+
RESPONSE=$(curl -s -X POST \
43+
-F token=${{ secrets.HPSF_GITLAB_TRIGGER_TOKEN }} \
44+
-F ref=development \
45+
-F "variables[GITHUB_PR_NUMBER]=${{ github.event.issue.number }}" \
46+
-F "variables[GITHUB_MERGE_REF]=refs/pull/${{ github.event.issue.number }}/merge" \
47+
-F "variables[GITHUB_TRIGGER_ACTOR]=${{ github.actor }}" \
48+
-F "variables[GITHUB_PR_TITLE]=${{ steps.setref.outputs.pr_title }}" \
49+
-F "variables[GITHUB_PIPELINE_NAME]=${GITHUB_PIPELINE_NAME}" \
50+
"https://gitlab.spack.io/api/v4/projects/${{ env.GITLAB_PROJECT_ID }}/trigger/pipeline")
51+
52+
echo "GitLab response: $RESPONSE"
53+
54+
PIPELINE_ID=$(echo "$RESPONSE" | jq -r '.id')
55+
56+
if [ "$PIPELINE_ID" == "null" ] || [ -z "$PIPELINE_ID" ]; then
57+
echo "Failed to trigger GitLab pipeline."
58+
exit 1
59+
fi
60+
61+
PIPELINE_URL="https://gitlab.spack.io/${{ env.GITLAB_PROJECT_PATH }}/-/pipelines/$PIPELINE_ID"
62+
echo "Pipeline URL: $PIPELINE_URL"
63+
echo "pipeline_url=$PIPELINE_URL" >> $GITHUB_OUTPUT
64+
65+
- name: Post status to GitHub PR
66+
shell: bash
67+
run: |
68+
PIPELINE_URL=${{ steps.trigger.outputs.pipeline_url }}
69+
COMMENT="GitLab CI has started at ${PIPELINE_URL}."
70+
71+
gh api \
72+
repos/${{ github.repository }}/issues/${{ github.event.issue.number }}/comments \
73+
-f body="$COMMENT" || echo "Failed to post comment to GitHub"
74+
env:
75+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitlab/README.md

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
This note describes the setup of our GitHub/GitLab integration. The main
2+
AMReX repo is hosted at https://github.com/AMReX-Codes/amrex, with a mirror
3+
at https://gitlab.spack.io/amrex/amrex. The GitLab mirror runs CI tests on
4+
runners hosted at
5+
https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers. We
6+
run three types of tests: nightly tests using the development branch, tests
7+
triggerd when new commits are pushed to the GitLab mirror (either through
8+
GitLab's regular pulling of the latest development branch from GitHub or by
9+
the developers directly), and tests triggered by a special comment from
10+
maintainers.
11+
12+
Instructions for creating a pull mirror can be found at
13+
https://docs.gitlab.com/user/project/repository/mirror/pull/. You can choose
14+
to pull selected branches or all branches. For AMReX, this choice does not
15+
matter since we only have one active branch. However, for repoositories with
16+
many active branches, pulling all branches may trigger CI jobs on every
17+
activity.
18+
19+
Instructions for creating schedules pipelines can be found at
20+
https://docs.gitlab.com/ci/pipelines/schedules/. By default, GitLab uses
21+
`.gitlab-ci.yml` at the repository root for pipeline configurations, but
22+
this can be changed. AMReX's GitLab CI file is in the `.gitlab`
23+
directory. To change the configuration file location, select `Settings ->
24+
CI/CD -> General pipelines` and update `CI/CD configuration file`.
25+
26+
CI/CD jobs triggered by push require no special setup. They can also be
27+
disabled.
28+
29+
For GitHub PR triggered jobs, we choose not to run automatically. Instead
30+
they need to be triggered by a `/run-hpsf-gitlab-ci` comment from authorized
31+
maintainers, which starts the `run-hpsf-gitlab-ci` workflow defined in
32+
`.github/workflows/trigger-hpsf-gitlab-ci.yml`. This workflow uses GitLab's
33+
REST API to start a pipeline job on GitLab. The PR comment triggered
34+
pipeline job uses the same configuration file shared with the scheduled
35+
pipeline: `.gitlab/hpsf-gitlab-ci.yml`. The PR comment triggered job pulls
36+
the PR branch from GitHub first before running tests. For this approach to
37+
work, we store a pipeline trigger (obtained from GitLab's `Settings -> CI/CD
38+
-> Pipeline trigger tokens`) as a secret at GitHub's `Settings -> Secrets
39+
and variables -> Actions -> Repository secrets`.
40+
41+
After the GitLab pipeline finishes, its `.post` stage will post the final
42+
status and a link to the results back to the GitHub PR as a comment. This is
43+
done through a GitHub App that we built and installed in the AMReX
44+
repository. The App was created via
45+
https://github.com/organizations/AMReX-Codes/settings/apps. It does not need
46+
"Webhook" access. For repository permissions, it only needs read & write to
47+
pull requests so it can create PR comments. The app requires a private key,
48+
which you generate during the setup. After the app was installed in the
49+
amrex repository, we got an installation ID. We then stored the app ID,
50+
installation ID and the private key in GitLab's `Settings -> CI/CD ->
51+
Variables`. The app ID isn't a secret. So you can store it as clear text. In
52+
fact, GitLab does not allow 7-digit masked variables anyway. The
53+
installation ID is also not senstive, but nevertheless we stored it as
54+
protected and masked. The private key is a secret that must be protected and
55+
masked. We also diabled "Expand" for all of these variables because the CI
56+
script doesn't need variable expansion. GitLab seems to have a bug that
57+
prevents saving the private key as a multi-line value, and saving it as a
58+
file didn't work either. So we encoded it with `base64 -w0` to turn it into
59+
a single line. That's why in the GitLab CI script we have to decode it.

.gitlab/hpsf-gitlab-ci.yml

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
workflow:
2+
name: '$GITHUB_PIPELINE_NAME'
3+
4+
.dependency-n-fetch-pr:
5+
stage: test
6+
before_script:
7+
- apt-get update && apt-get install -y git cmake python3 libopenmpi-dev openmpi-bin
8+
- |
9+
if [ -n "$GITHUB_PR_NUMBER" ]; then
10+
echo "GITHUB_PR_NUMBER=$GITHUB_PR_NUMBER"
11+
echo "GITHUB_MERGE_REF=$GITHUB_MERGE_REF"
12+
echo "GITHUB_TRIGGER_ACTOR=$GITHUB_TRIGGER_ACTOR"
13+
echo "GITHUB_PR_TITLE=$GITHUB_PR_TITLE"
14+
git remote remove github || true
15+
git remote add github https://github.com/amrex-codes/amrex.git
16+
git fetch --depth=1 github "${GITHUB_MERGE_REF}"
17+
git checkout FETCH_HEAD
18+
fi
19+
20+
Nvidia-H100-single-precision:
21+
extends: .dependency-n-fetch-pr
22+
tags: [nvidia-h100]
23+
image: nvcr.io/nvidia/cuda:12.8.0-devel-ubuntu24.04
24+
script:
25+
- |
26+
cmake -S . -B build \
27+
-DCMAKE_VERBOSE_MAKEFILE=ON \
28+
-DAMReX_PRECISION=SINGLE \
29+
-DAMReX_PARTICLES_PRECISION=SINGLE \
30+
-DAMReX_SPACEDIM="1;2;3" \
31+
-DAMReX_FFT=ON \
32+
-DAMReX_EB=ON \
33+
-DAMReX_ENABLE_TESTS=ON \
34+
-DAMReX_FORTRAN=OFF \
35+
-DAMReX_GPU_BACKEND=CUDA \
36+
-DCMAKE_C_COMPILER=$(which gcc) \
37+
-DCMAKE_CXX_COMPILER=$(which g++) \
38+
-DCMAKE_CUDA_HOST_COMPILER=$(which g++) \
39+
-DAMReX_CUDA_ARCH="9.0" \
40+
-DAMReX_CUDA_ERROR_CROSS_EXECUTION_SPACE_CALL=ON \
41+
-DAMReX_CUDA_ERROR_CAPTURE_THIS=ON
42+
- cmake --build build -j 16
43+
- export OMPI_ALLOW_RUN_AS_ROOT=1
44+
- export OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
45+
- export AMREX_THE_ARENA_INIT_SIZE=1e9
46+
- ctest -j 2 --test-dir build --output-on-failure
47+
48+
AMD-MI300A:
49+
extends: .dependency-n-fetch-pr
50+
tags: [amd-mi300]
51+
image: rocm/dev-ubuntu-24.04:6.4.3-complete
52+
script:
53+
- export ROCM_PATH=/opt/rocm-6.4.3
54+
- export PATH=$ROCM_PATH/bin:$ROCM_PATH/llvm/bin:$PATH
55+
- export LD_LIBRARY_PATH=$ROCM_PATH/lib:$ROCM_PATH/lib64:$LD_LIBRARY_PATH
56+
- export HIP_PATH=$ROCM_PATH
57+
- export HIP_CLANG_PATH=$ROCM_PATH/llvm/bin
58+
- export CMAKE_PREFIX_PATH=$ROCM_PATH:$CMAKE_PREFIX_PATH
59+
- |
60+
cmake -S . -B build \
61+
-DCMAKE_VERBOSE_MAKEFILE=ON \
62+
-DAMReX_SPACEDIM="1;2;3" \
63+
-DAMReX_FFT=ON \
64+
-DAMReX_EB=ON \
65+
-DAMReX_ENABLE_TESTS=ON \
66+
-DAMReX_FORTRAN=OFF \
67+
-DAMReX_GPU_BACKEND=HIP \
68+
-DAMReX_AMD_ARCH=gfx942 \
69+
-DCMAKE_C_COMPILER=$(which clang) \
70+
-DCMAKE_CXX_COMPILER=$(which clang++) \
71+
-DCMAKE_CXX_STANDARD=17
72+
- cmake --build build -j 16
73+
- export OMPI_ALLOW_RUN_AS_ROOT=1
74+
- export OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
75+
- export AMREX_THE_ARENA_INIT_SIZE=1e9
76+
- ctest -j 2 --test-dir build --output-on-failure
77+
78+
Intel-PVC:
79+
extends: .dependency-n-fetch-pr
80+
tags: [intel-data-center-max-1100]
81+
image: intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04
82+
script:
83+
- sycl-ls
84+
- export ONEAPI_DEVICE_SELECTOR=level_zero:gpu
85+
- |
86+
cmake -S . -B build \
87+
-DCMAKE_VERBOSE_MAKEFILE=ON \
88+
-DAMReX_MPI=OFF \
89+
-DAMReX_SPACEDIM="1;2;3" \
90+
-DAMReX_FFT=ON \
91+
-DAMReX_EB=ON \
92+
-DAMReX_ENABLE_TESTS=ON \
93+
-DAMReX_FORTRAN=OFF \
94+
-DAMReX_GPU_BACKEND=SYCL \
95+
-DCMAKE_C_COMPILER=$(which icx) \
96+
-DCMAKE_CXX_COMPILER=$(which icpx) \
97+
-DCMAKE_CXX_FLAGS="-fp-model=precise" \
98+
-DAMReX_PARALLEL_LINK_JOBS=8
99+
- cmake --build build -j 16
100+
- export AMREX_THE_ARENA_INIT_SIZE=1e9
101+
- ctest -j 4 --test-dir build --output-on-failure
102+
103+
post_github_comment:
104+
stage: .post
105+
image: python:3.11-slim
106+
script:
107+
- pip install PyJWT requests cryptography
108+
- |
109+
python3 << 'EOF'
110+
import jwt
111+
import time
112+
import requests
113+
import os
114+
import sys
115+
import base64
116+
117+
# --- START: Get effective Pipeline Status from GitLab API ---
118+
gitlab_api_url = os.environ['CI_API_V4_URL']
119+
project_id = os.environ['CI_PROJECT_ID']
120+
pipeline_id = os.environ['CI_PIPELINE_ID']
121+
job_token = os.environ['CI_JOB_TOKEN']
122+
current_job_name = os.environ['CI_JOB_NAME']
123+
124+
final_status = 'success' # Assume success until a failure is found
125+
try:
126+
# Get all jobs for the pipeline
127+
api_endpoint = f"{gitlab_api_url}/projects/{project_id}/pipelines/{pipeline_id}/jobs"
128+
headers = {'JOB-TOKEN': job_token}
129+
params = {'scope[]': ['success', 'failed', 'canceled'], 'per_page': 100}
130+
response = requests.get(api_endpoint, headers=headers, params=params)
131+
response.raise_for_status()
132+
133+
jobs = response.json()
134+
for job in jobs:
135+
# We only care about the result of test/build jobs, not this notification job itself
136+
if job['name'] == current_job_name:
137+
continue
138+
139+
if job['status'] in ['failed', 'canceled']:
140+
final_status = 'failed'
141+
break # Found a failure, no need to check further
142+
143+
except requests.exceptions.RequestException as e:
144+
print(f"Error fetching job statuses from GitLab API: {e}", file=sys.stderr)
145+
final_status = 'unknown' # If API fails, report unknown status
146+
# --- END: Get effective Pipeline Status ---
147+
148+
# Generate JWT for GitHub App authentication
149+
app_id = os.environ['GITHUB_APP_ID']
150+
payload = {
151+
'iat': int(time.time()),
152+
'exp': int(time.time()) + 600,
153+
'iss': app_id
154+
}
155+
token = jwt.encode(
156+
payload,
157+
base64.b64decode(os.environ['GITHUB_APP_PRIVATE_KEY']).decode('utf-8'),
158+
algorithm='RS256'
159+
)
160+
161+
# Get installation access token
162+
installation_id = os.environ['GITHUB_APP_INSTALLATION_ID']
163+
token_response = requests.post(
164+
f'https://api.github.com/app/installations/{installation_id}/access_tokens',
165+
headers = {
166+
'Authorization': f'Bearer {token}',
167+
'Accept': 'application/vnd.github+json'
168+
}
169+
)
170+
if token_response.status_code != 201:
171+
print(f"Failed to get access token: HTTP {token_response.status_code}")
172+
sys.exit(1)
173+
174+
# Post comment to PR
175+
pr_number = os.environ.get('GITHUB_PR_NUMBER')
176+
pipeline_url = os.environ['CI_PIPELINE_URL']
177+
178+
comment_body = f"GitLab CI {pipeline_id} finished with status: **{final_status}**. See details at {pipeline_url}."
179+
180+
comment_response = requests.post(
181+
f'https://api.github.com/repos/AMReX-Codes/amrex/issues/{pr_number}/comments',
182+
headers = {
183+
'Authorization': f'Bearer {token_response.json()["token"]}',
184+
'Accept': 'application/vnd.github+json'
185+
},
186+
json={'body': comment_body}
187+
)
188+
if comment_response.status_code != 201:
189+
print(f"Failed to post comment: HTTP {comment_response.status_code}")
190+
sys.exit(1)
191+
print("Successfully posted comment to GitHub PR")
192+
EOF
193+
when: always
194+
rules:
195+
- if: $GITHUB_PR_NUMBER

Docs/sphinx_documentation/source/GPU.rst

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -560,8 +560,16 @@ to :cpp:`The_Arena()` to reduce memory fragmentation.
560560
In :cpp:`amrex::Initialize`, a large amount of GPU device memory is
561561
allocated and is kept in :cpp:`The_Arena()`. The default is 3/4 of the
562562
total device memory, and it can be changed with a :cpp:`ParmParse`
563-
parameter, ``amrex.the_arena_init_size``, in the unit of bytes. The default
564-
initial size for other arenas is 8388608 (i.e., 8 MB). For
563+
parameter, ``amrex.the_arena_init_size``, in the unit of bytes. The default
564+
can also be changed with an environment variable
565+
``AMREX_THE_ARENA_INIT_SIZE=X``, where ``X`` is the number of bytes. When
566+
both the :cpp:`ParmParse` parameter and the environment variable are
567+
present, the former will override the latter. In both cases, the number
568+
string could have optional single quotes ``'`` as separators (e.g.,
569+
``10'000'000'000``). It may also use floating-point notation (``2.5e10``),
570+
as long as converting it does not introduce any loss of precision.
571+
572+
The default initial size for other arenas is 8388608 (i.e., 8 MB). For
565573
:cpp:`The_Managed_Arena()` and :cpp:`The_Device_Arena()`, it can be changed
566574
with ``amrex.the_managed_arena_init_size`` and
567575
``amrex.the_device_arena_init_size``, respectively, if they are not an alias

0 commit comments

Comments
 (0)