Add amrex::Gpu::streamFree

[AlexanderSinn]

Summary

This PR adds the function amrex::Gpu::streamFree (Arena* arena, void* mem) that can be used to free memory the next time the current GPU stream is synchronized.

This is based on #4432 but with much reduced complexity from OMP.
The interface is now opt-in and always available, instead of needing to be enabled using runtime parameters.

Additional background

Checklist

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• include documentation in the code and/or rst files, if appropriate

[WeiqunZhang]

/run-hpsf-gitlab-ci

[github-actions]

GitLab CI has started at https://gitlab.spack.io/amrex/amrex/-/pipelines/1319922.

[amrex-gitlab-ci-reporter]

GitLab CI 1319922 finished with status: success. See details at https://gitlab.spack.io/amrex/amrex/-/pipelines/1319922.

[AlexanderSinn]

Maybe this should be named asyncFree or freeAsync instead.

[WeiqunZhang]

I like freeAsync, which is similar to copyAsync.

[AlexanderSinn]

For a case with very few particles, this helps to significantly improve the performance of hipace shiftSlippedParticles (which calls amrex::partitionParticles) by removing two stream synchronizes.

AMReX dev, 79 µs on the GPU:

Using this PR, 59 µs on the GPU: