Fix GPU restart for pure SoA particles

[atmyers]

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• include documentation in the code and/or rst files, if appropriate

[ax3l]

Thank you for debugging and fixing this! ✨

There are a few CI test issues left:

0::Assertion `sm_old == sm_new' failed, file "/Users/runner/work/amrex/amrex/Tests/Particles/CheckpointRestartSOA/main.cpp", line 174 !!!

and

1::Assertion `ptemp.id() > 0' failed, file "/Users/runner/work/amrex/amrex/Src/Particle/AMReX_ParticleIO.H", line 1022 !!!

overlooked CI

[WeiqunZhang]

For Nyx restart, the current development branch works on CPU and GPU. With this branch, it works on CPU, but fails on GPU.

https://ccse.lbl.gov/pub/GpuRegressionTesting/Nyx/2024-03-02/index.html

[atmyers]

I just fixed some things, hopefully everything will pass now.

[WeiqunZhang]

The Nyx tests pass now. But this IAMR non-restart test still fails. https://ccse.lbl.gov/pub/GpuRegressionTesting/IAMR/2024-03-02-001/Part-2d.html

[WeiqunZhang]

The reconnection GPU tests also fail. But the problem might be in the reconnection code, not amrex.

This PR: https://ccse.lbl.gov/pub/GpuRegressionTesting/reconnection/2024-03-02/index.html

development: https://ccse.lbl.gov/pub/GpuRegressionTesting/reconnection/2024-03-01/index.html
The restart test started Friday morning was never able to finish, and I manually killed it today.

@RevathiJambunathan

[WeiqunZhang]

The amrex ParticleMesh test fails. https://ccse.lbl.gov/pub/GpuRegressionTesting/AMReX/2024-03-02/ParticleMesh.html

[WeiqunZhang]

For reconnection, I also tried an earlier version (that works with the development branch of amrex) with this PR, the errors were similar to those of the latest reconnection code plus this PR. https://ccse.lbl.gov/pub/GpuRegressionTesting/reconnection/2024-03-02-001/index.html

[RevathiJambunathan]

Update : I do see the illegal access memory with this current PR, when I ran with 2 GPUs - same as https://ccse.lbl.gov/pub/GpuRegressionTesting/reconnection/2024-03-02-001/Baseline2Dsigma30Perturbation.html

1 . The test where I compiled with the dev branch crashed at the first checkpoint I/O, and this PR fixes that issue

2. However, when I run the current PR, the test still crashes. Now it write the checkpoint file, but crashes at RedistributeLocal(). Backtrace points to

   amrex/Src/Particle/AMReX_ParticleUtil.H

   Line 588 in 3525b4a

   partitionParticlesByDest (PTile& ptile, const PLocator& ploc, CellAssignor&& assignor,

   and

   amrex/Src/Particle/AMReX_ParticleUtil.H

   Line 647 in 3525b4a

   return partitionParticles(ptile,

[AlexanderSinn]

The issue Reva is seeing was fixed in #3769, the branch of this PR is outdated and hasn't got that fix yet.

[WeiqunZhang]

The reconnection code works now.

[WeiqunZhang]

Remaining issues:

• https://ccse.lbl.gov/pub/GpuRegressionTesting/AMReX/2024-03-04-001/ParticleMesh.html
• https://ccse.lbl.gov/pub/GpuRegressionTesting/IAMR/2024-03-04-001/Part-2d.html

Or is the old behavior actually wrong?

[WeiqunZhang]

The issue is in the GPU version of packIOData. If I use managed memory and force it to use the CPU version of packIOData, it works.

[WeiqunZhang]

There might be another issue in the GPU version of packIOData besides the one that causes the regression failure. The IntVector offsets computed by exclusive sum is never used. If the particle io flags is not all true, that will not be right.

[RevathiJambunathan]

@AlexanderSinn @WeiqunZhang Thanks!
Yes, updating branch now!

[WeiqunZhang]

To summarize the current status, all regression tests I have tried have passed (including the reconnection) except for a few tests in amrex and IAMR after the development branch was merged into this. (Yes, I merged and pushed to this PR.) The amrex and IAMR regressions issues are new issues introduced in this PR. The code change suggestion I made above seems to solve the new regression issues. (No, I did not commit it). There is an existing defect regarding the filtering flags. That should be easy to fix either in this PR or a follow-up.

[WeiqunZhang]

To clarify, the reconnection tests work with the current PR and I can approve this PR once the new IAMR and amrex issued are resolved.

[ax3l]

Just checking for my understanding:
Did WarpX restarts pass now? :)

There is an existing defect regarding the filtering flags. That should be easy to fix either in this PR or a follow-up.

Is that fixed in the merge or to do?

[atmyers]

The WarpX restart issues seem to pass now based on the minimal reproducer. The offsets bug is still to-do.