Update particle docs for expanded idcpu range (#4758)

I am making a series of PRs that update the particle documentation. This
first one explains the current convention re: particle id and cpu
numbers.

The proposed changes:
- [ ] fix a bug or incorrect behavior in AMReX
- [ ] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX
users
- [x] include documentation in the code and/or rst files, if appropriate

---------

Co-authored-by: Axel Huebl <[email protected]>

Author: atmyers

Docs/sphinx_documentation/source/Particle.rst

@@ -24,24 +24,41 @@ type of components that the particles carry. The first template parameter is the
 number of extra :cpp:`Real` variables this particle will have (either single or
 double precision [1]_), while the second is the number of extra integer
 variables.  It is important to note that this is the number of *extra* real and
-integer variables; a particle will always have at least :cpp:`BL_SPACEDIM` real
-components that store the particle's position and 2 integer components that
-store the particle's :cpp:`id` and :cpp:`cpu` numbers. [2]_
-
-The particle struct is designed to store these variables in a way that
-minimizes padding, which in practice means that the :cpp:`Real` components
-always come first, and the integer components second. Additionally, the
-required particle variables are stored before the optional ones, for both the
-real and the integer components. For example, say we want to define a particle
-type that stores a mass, three velocity components, and two extra integer
-flags. Our particle struct would be set up like:
+integer variables; a particle will always have at least :cpp:`AMREX_SPACEDIM` real
+components that store the particle's position and one 64-bit integer component that
+stores a combination of the particle's unique `id` and `cpu` numbers.
+
+The particle struct is stored such that the :cpp:`Real` components come first
+and the integer component second. Additionally, the required particle variables
+are stored before the optional ones, for both the real and the integer components.
+For example, say we want to define a particle type that stores a mass, three velocity
+components, and two extra integer flags. Our particle struct would be declared like:
 
 ::
 
       Particle<4, 2> p;
 
-and the order of the particle components in would be (assuming :cpp:`BL_SPACEDIM` is 3):
-:cpp:`x y z m vx vy vz id cpu flag1 flag2`.  [3]_
+and the order of the particle components in would be (assuming :cpp:`AMREX_SPACEDIM` is 3):
+:cpp:`x y z m vx vy vz idcpu flag1 flag2`.  [2]_
+
+The `idcpu` variable stores a combination of the MPI process a particle was generated on
+(the `cpu`) and an identification number that is specific to that process (the `id`).
+The combination of these numbers is unique across processes. This is done to facilitate
+the creation of particle initial conditions in parallel. In storing these
+identifying numbers, 39 bits are devoted to the `id`, allowing approximately 550 billion
+possible *local* `id` numbers, and 24 bits are used to store the `cpu`, allowing about 16.8 million
+unique (MPI) processes.
+One bit is devoted to mark a particle valid or invalid.
+
+To pack and unpack these numbers, one uses the following syntax:
+
+::
+
+       Particle <0, 0> p;
+       p.id() = 1;
+       p.cpu() = 0;
+       amrex::Print() << p.m_idcpu << "\n"; // 9223372036871553024
+       amrex::Print() << p.id() << " " << p.cpu() << "\n"; // 1 0
 
 Setting Particle data
 ---------------------
@@ -440,8 +457,7 @@ example:
            real(amrex_particle_real) :: pos(3)
            real(amrex_particle_real) :: vel(3)
            real(amrex_particle_real) :: acc(3)
-           integer(c_int)   :: id
-           integer(c_int)   :: cpu
+           integer(c_int64_t) :: idcpu
         end type particle_t
 
 is equivalent to a particle struct you get with :cpp:`Particle<6, 0>`. Here,
@@ -793,9 +809,6 @@ The following runtime parameters affect the behavior of virtual particles in Nyx
    Particles default to double precision for their real data. To use single precision, compile your code with ``USE_SINGLE_PRECISION_PARTICLES=TRUE``.
 
 .. [2]
-   Note that :cpp:`cpu` stores the number of the process the particle was *generated* on, not the one it's currently assigned to. This number is set on initialization and never changes, just like the particle :cpp:`id`. In essence, the particles have two integer id numbers, and only the combination of the two is unique. This was done to facilitate the creation of particle initial conditions in parallel.
-
-.. [3]
    Note that for the extra particle components, which component refers to which
    variable is an application-specific convention - the particles have 4 extra real comps, but which one is "mass" is up
    to the user. We suggest using an :cpp:`enum` to keep these indices straight; please

Docs/sphinx_documentation/source/Particle_Chapter.rst

@@ -3,16 +3,15 @@
 Particles
 =========
 
-In addition to the tools for working with mesh data described in previous
-chapters, AMReX also provides data structures and iterators for performing
-data-parallel particle simulations. Our approach is particularly suited to
-particles that interact with data defined on a (possibly adaptive)
-block-structured hierarchy of meshes. Example applications include
-Particle-in-Cell (PIC) simulations, Lagrangian tracers, or particles that exert
-drag forces onto a fluid, such as in multi-phase flow calculations. The overall
-goals of AMReX's particle tools are to allow users flexibility in specifying
-how the particle data is laid out in memory and to handle the parallel
-communication of particle data. In the following sections, we
+In addition to the tools for working with mesh data described in previous chapters,
+AMReX provides data structures and iterators for performing data-parallel particle simulations.
+While these tool can be used to implement pure particle methods, they are particularly
+suited for methods in which particles interact with data defined on mesh or
+hierarchy of meshes. Example applications include Particle-in-Cell (PIC) simulations,
+Lagrangian tracers, and particles that exert drag forces onto a fluid. The overall
+goals of AMReX's particle tools are to allow users to express a variety of useful
+operations on particle data, including particle-mesh and particle-particle operations,
+in a performance-portable and scalable manner. In the following sections, we
 give an overview of AMReX's particle classes and how to use them.