Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce particle shape when a particle approaches the EB #5209

Conversation

oshapoval
Copy link
Member

@oshapoval oshapoval commented Sep 3, 2024

Description edited by @RemiLehe

Overview

This PR reduces the particle shape to order 1, when the particle gets closer to the embedded boundary:
Screenshot 2025-01-23 at 8 46 34 AM

This ensures that the particle does not deposit charge in valid cells, at the time when it is removed, which in turn ensures proper charge conservation with the electromagnetic solver.

Implementation

  • This PR allocates and initializes a new mask eb_reduce_particle_shape (and iMultiFab) that indicates in which cells to reduce the particle shape.
  • The deposition kernels have been modified to use this flag. In order to make sure that this PR does not affect the performance of the deposition kernel in the absence of EB, two versions of the deposition kernel are compiled.

Tests

This PR adds tests similar to the ones introduced in #5562 to check for charge conservation near the embedded boundary, but with higher-order shape factors:

  • The 2D tests fail on development for shape 2 and 3 but pass on this PR.
  • For some reason, the 3D and RZ tests only fail on development for shape 3 ; they do pass for this PR. It is not clear why the tests do not fail on development with shape 2.

Note: For now, this PR only modifies the current deposition (and only the Esirkepov kernel). A follow-up PR will also modify the charge deposition.

oshapoval and others added 30 commits April 12, 2024 12:19
@RemiLehe RemiLehe changed the title [WIP] Reduce particle shape when a particle approaches the EB Reduce particle shape when a particle approaches the EB Jan 24, 2025
Copy link
Member

@ax3l ax3l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thank you for this! I added inline comments for small details.

I was wondering if it might help to split a the doEsirkepovDepositionShapeN function up into inlined sub-functions, in case it gets too crowded.

}

}

ComputeDistanceToEB();
MarkReducedShapeCells( m_eb_reduce_particle_shape[lev], eb_fact, WarpX::nox );
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you don't need to pass m_eb_reduce_particle_shape (or even all) arguments here and could call it like ComputeDistanceToEB(). (Because both are member functions of the same class.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good.
Part of the motivation here is to maybe do a follow-up PR where some of the Mark...Cells functions become free functions.


// Every cell in box is regular: do not reduce particle shape in any cell
amrex::ParallelFor(box, [=] AMREX_GPU_DEVICE (int i, int j, int k) {
eb_reduce_particle_shape_arr(i, j, k) = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default from setVal, but ok to be explicit.

@@ -656,7 +659,10 @@ void doEsirkepovDepositionShapeN (const GetParticlePosition<PIdx>& GetPosition,
const amrex::XDim3 & xyzmin,
amrex::Dim3 lo,
amrex::Real q,
[[maybe_unused]]int n_rz_azimuthal_modes)
[[maybe_unused]]int n_rz_azimuthal_modes,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not important formatting of existing code:

Suggested change
[[maybe_unused]]int n_rz_azimuthal_modes,
[[maybe_unused]] int n_rz_azimuthal_modes,

Source/Particles/Deposition/CurrentDeposition.H Fixed Show resolved Hide resolved
// If particle is close to the embedded boundary, recompute deposition with order 1 shape
if constexpr (reduce_shape_control == has_reduced_shape) {
if (reduce_shape_new) {
for (int i=0; i<depos_order+3; i++) sx_new[i] = 0.; // Erase previous deposition
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a look on godbolt and the compiler seems to unroll this constant-size loop, so no need to write it more obscurely.

for (int i=0; i<depos_order+3; i++) sx_old[i] = 0.; // Erase previous deposition
compute_shifted_shape_factor_order1( sx_old+depos_order/2, x_old, i_new+depos_order/2 ); // Redeposit with order 1
}
// Note: depos_order/2 in the above code corresponds to the shift between the index of the lowest point
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment, very helpful to parse this.
If it helps to name this, you could also write this in an const int before the function call.

@RemiLehe RemiLehe enabled auto-merge (squash) January 27, 2025 22:36
@RemiLehe RemiLehe merged commit 0b10fca into ECP-WarpX:development Jan 28, 2025
37 checks passed
@oshapoval
Copy link
Member Author

oshapoval commented Jan 29, 2025

The performance of CurrentDeposition was tested by running a uniform plasma test (input attached below) case three times, on Perlmutter.
Note: WarpX was compiled from the latest dev branch (commit gcb30300cbba1, feature branch is already merged) and earlier dev branch (commit g4cae61128ee7) with the following options:
cmake -S . -B build -DWarpX_OPENPMD=ON -DWarpX_DIMS=3 -DWarpX_COMPUTE=CUDA

Results: feature branch does not change performance

latest dev branch (commit gcb30300cbba1), EB
Run # Name                                                        NCalls  Excl. Min  Excl. Avg  Excl. Max   Max %
run1     WarpXParticleContainer::DepositCurrent::CurrentDeposition     200      5.083      5.087      5.089  46.80%
run2     WarpXParticleContainer::DepositCurrent::CurrentDeposition     200      5.084      5.087      5.089  46.53%
run3     WarpXParticleContainer::DepositCurrent::CurrentDeposition     200      5.087      5.089      5.092  46.56%   


earlier dev branch (commit g4cae61128ee7), EB
Run # Name                                                        NCalls  Excl. Min  Excl. Avg  Excl. Max   Max %
run1     WarpXParticleContainer::DepositCurrent::CurrentDeposition     200      5.063      5.066      5.072  46.74%
run2     WarpXParticleContainer::DepositCurrent::CurrentDeposition     200      5.059      5.064      5.068  46.54%
run3     WarpXParticleContainer::DepositCurrent::CurrentDeposition     200      5.072      5.077      5.08   46.33%

EB was set via implicit function:
my_constants.R = 45.e-6 warpx.eb_implicit_function = "(x**2 + y**2 - R**2)""

input.txt

@ax3l ax3l mentioned this pull request Feb 4, 2025
amrex::Array4<int> const & eb_reduce_particle_shape_arr = eb_reduce_particle_shape->array(mfi);

// Check if the box (including one layer of guard cells) contains a mix of covered and regular cells
const amrex::Box& eb_info_box = mfi.tilebox(amrex::IntVect::TheCellVector()).grow(1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Little bug: fix in #5635

ax3l added a commit that referenced this pull request Feb 4, 2025
Follow-up to #5209: My compiler says those locations would reference
temporary objects that were destroyed at the end of the line. That seems
to be the case indeed.

Copy instead to make the temporary a named and thus persistent variable.

![Screenshot from 2025-02-03
16-59-22](https://github.com/user-attachments/assets/8259f6d7-099b-4d09-8382-f24baefb5793)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: boundary PML, embedded boundaries, et al.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants