ADIOS2 v2.7.1: Spack Build on Summit Breaks IB #2836
Comments
|
Well, the initial error, "Number of active IB device ports not supported" is unfamiliar to either me or google. I guess I don't know whether or not to credit the fewer dependencies of warpx and libadios2_cxx11_mpi.so as the root cause or not. Seems like those dependencies are usually transitive, but I don't know if spack does something funny there. If I were guessing though I might guess that there might be a version conflict with whatever MPI relies upon for RDMA and whatever we're using. @chuckatkins , maybe something like we resolved on rhea by loading the rdma-core module? |
ax3l
added a commit
to ECP-WarpX/WarpX
that referenced
this issue
Aug 20, 2021
Tickets open: - OLCFHELP-3319 - ornladios/ADIOS2#2836
|
I think that error originates from the bowels of SpectrumMPI, but likely triggered due to a dependency mismatch. For what it's worth, this is the WarpX and that's the list with ADIOS2 (44 direct loads): |
|
This is a known problem on summit and a mismatch between the rdma-core libraries used by spectrum-mpi and spack's libfabric. I believe the admins built a specific module for us to fix this. Try using the libfabric/1.7.0-sysrdma module instead of the default libfabric. |
|
But we can also try to reproduce this with https://github.com/ornladios/ADIOS2-Examples if you like. In an interactive session via I can see the same issues. |
|
Just to clarify @ax3l, did you try ADIOS2-Examples with the libfabric/1.7.0-sysrdma as Chuck suggested and still see a failure? |
|
Not yet, I posted at the same time and then went in the weekend :)
…On August 21, 2021 7:37:09 AM PDT, Greg Eisenhauer ***@***.***> wrote:
Just to clarify @ax3l, did you try ADIOS2-Examples with the libfabric/1.7.0-sysrdma as Chuck suggested and still see a failure?
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#2836 (comment)
|
|
Thanks a lot for the quick suggestion @chuckatkins! If a custom Summit-specific libfabric is the solution: maybe the OLCF E4S team can register this version of libfabric as an external spack package for all their builds. That way, all packages built out use that specifically tweaked libfabric that you mentioned. (Found this config but seems outdated.) Just curious: is that an IBM or Mellanox fork of libfabric on Summit? Or were the changes/fixes upstreamed? |
An easy workaround should be to just use $ export LD_PRELOAD="/usr/lib64/libibverbs.so.1:/usr/lib64/librdmacm.so.1"
The problem is a mismatch in the |
|
@chuckatkins @ax3l Following up on the OLCFHELP-3319 ticket, we put a module |
roelof-groenewald
added a commit
to ModernElectron/WarpX
that referenced
this issue
Aug 25, 2021
* AMReX/PICSAR: Weekly Update (ECP-WarpX#2199) Weekly update to latest AMReX. No changes in PICSAR. * do_pml should not be parsed anymore. (ECP-WarpX#2183) * do_pml not parsed. remove code that was added to support both types of boundary interface * add paranthesis * fix eol and move to BackwardCompatibility * missing semicolon * Update Source/WarpX.cpp * clean input files in examples * delete do_pml in performance test input * fixing an example input file in docs * Update Source/WarpX.cpp Co-authored-by: Axel Huebl <[email protected]> * cleaning up docs * Update Docs/source/usage/parameters.rst * fix eol Co-authored-by: Axel Huebl <[email protected]> * Multi-J (Hybrid): Fix Bug with Current Centering (ECP-WarpX#2181) * Use less guard cells in ParallelCopy of refined level's UpdateAuxilaryData (ECP-WarpX#2144) * Use less guard cells in ParallelCopy of refined level's UpdateAuxilaryData * Update Source/Parallelization/WarpXComm.cpp * Add inline comments and use explicit type for ng_src Co-authored-by: Edoardo Zoni <[email protected]> * Update Scripts: .ini file (ECP-WarpX#2191) Update regression tester .ini files as well in weekly/release updates of AMReX and PICSAR. * Minor fix to the documentation of the plasma lens (ECP-WarpX#2200) * CI: Cancel Prev. GH Action on Push (ECP-WarpX#2202) Save CI resources by canceling already running or waiting builds if a PR is updated. This was previously only done for Azure and now also for GH actions. * openPMD-api: 0.14.2 (ECP-WarpX#2150) This updates our requirements to openPMD-api 0.14.2+ Among others, this release introduced resizable data sets, which we will soon use for particle backtransformed diagnostics. * update use-sensei flag (ECP-WarpX#2192) * update use-sensei flag also add FlushFormatSensei.cpp to CMakeLists * CMake: WarpX SENSEI Option Co-authored-by: Axel Huebl <[email protected]> * Rho Diags: Do Not Allocate `rho_fp/cp`, Remove `plot_rho`, `setplot_rho` (ECP-WarpX#2203) * Bug fixed: looping over all species now within `OneStep_multiJ` (ECP-WarpX#2207) * Bug fixed: looping over all species now within 'OneStep_multij' * Bug fixed: looping over all laser particles within DepositCharge in multi-J * Implemented the parsing of integer input parameters (ECP-WarpX#2138) * Implemented the integer parser * Updated comment * Updated documentation * Fixed unused parameters * Added some additional documentation * Reworked the implementation so that expressions are evaluated as real and rounded to the nearest integer * Fixed loop type * Copied over initial value of variable to the real instance * Update Source/Utils/WarpXUtil.cpp make result const * Update Source/Utils/WarpXUtil.cpp make result const * Update Source/Utils/WarpXUtil.cpp Fix comment * Added safeCastToInt * Fixed adding of safeCastToInt * Cleaned up safe casting routine * Added parsing of more integer inputs * Cleaned up the integer parser, removing unneeded cast from int to real * Made x a const in safeCastToInt Co-authored-by: Neïl Zaim <[email protected]> Co-authored-by: Axel Huebl <[email protected]> * Ionization.H: Fix Bug (Division by Zero) (ECP-WarpX#2214) * Add plot_raw_rho in parameters doc (ECP-WarpX#2212) * add plot_raw_rho in parameters doc * Update Docs/source/usage/parameters.rst Co-authored-by: Luca Fedeli <[email protected]> Co-authored-by: Luca Fedeli <[email protected]> * Docs: Summit with RHEL8 (ECP-WarpX#2216) Summit has undergone a major software update to RHEL8. The default compilers and CUDA version have been modernized, among others providing C++17 support by default. Also, our scientific I/O stack is now system-provided, thanks to our Spack and E4S efforts 🎉 Please update your `warpx.profile` on Summit, re-build your Python virtual environment and re-compile your executables. * Improved error handling when the libwarpx shared object library can't be loaded (ECP-WarpX#2215) * Improved error handling when the libwarpx shared library can't be loaded * Removed extra newline * Improved check and ended program on error * Particle boundary scrape (ECP-WarpX#2124) * Some prelimary refactoring. * missing header * implement scraping particles that leave the domain boundaries into buffers * fix tabs * missing return * merging * remove redefinition * functor to work around cuda bug. * handle 2D * Add support for EB buffer * protect for AMREX_USE_EB, static_assert that EB and RZ aren't both on. * fix unused * add inputs file * add test * fix bugs, remove print * fix test * fix test path. * remove no-op code * adding clear particles method * attempt at adding time stamp * Use integer step number instead of physical time to timestamp particles; also put shared code into named functor. * move call to before apply boundary conditions * use more descriptive inputs parameter * Update Source/Particles/ParticleBoundaryBuffer.cpp * fix comp bug * move CopyAndTimestamp to cpp file * also move IsOutsideProblemDomain functor * Rename to m_particle_boundary_buffer - Name: currently only used for boundary scraping - Singular: only one instance * Fix missing EOF newline * Typo: author * Param Read / Init: Cleanup For Simplify Co-authored-by: Roelof Groenewald <[email protected]> Co-authored-by: Axel Huebl <[email protected]> * Updated the installation instructions (ECP-WarpX#2218) * Docs: OLCF ADIOS2 Currently Broken Tickets open: - OLCFHELP-3319 - ornladios/ADIOS2#2836 * Allow Silver-Mueller boundary conditions to only be applied in certain directions (ECP-WarpX#2220) * Apply Silver-Mueller boundary only in requested direction * Include a test with independent Silver-Mueller boundary conditions * Correct typo in implementation * Added transform of fields from lab to boosted frame (ECP-WarpX#2201) * Evolve: Reorder py_afterstep and cur_time break (ECP-WarpX#2213) * Docs: Cori PICMI Instructions (ECP-WarpX#2219) * Docs: Cori PICMI Instructions Too tricky to get right to just cross-link - add documentation how to build & run PICMI interfaces on Cori. * Link: Jupyter Instructions * No particle resorting when no species (ECP-WarpX#2136) * default sort interval for particles if there are not species * declare parmparse Co-authored-by: Edoardo Zoni <[email protected]> * fix eol * define and set sort_interval default to -1 and then reset them to 4 for GPU, -1 for CPU if there are species/lasers in the input Co-authored-by: Edoardo Zoni <[email protected]> * AMReX/PICSAR: Weekly Update (ECP-WarpX#2222) Weekly update to latest AMReX. No changes in PICSAR. * Scalar field interpolator from grid to particle position (ECP-WarpX#2221) * refactored distance to EB calculation to have a more general function to interpolate from a scalar field on the grid to a given position * changed RZ error message to general interpolation function * changed function names to specify that scalar field interpolation is for a nodal field only * Check for unused WarpX environment variables when compiling (ECP-WarpX#2208) * Added check for unused 'WarpX' environment variables * Changed the approach of this - Go through environment variables as they are used to set cmake flags, and then check any remaining if they start with warpx. * Improved the warning message Co-authored-by: Axel Huebl <[email protected]> * Ascent/SENSEI: Add Profiler (ECP-WarpX#2204) * Ascent/SENSEI: Add Profiler Add overall write profilers to Ascent & SENSEI. Add detailed profile for sub-operaions (blueprint, publish, execute) to Ascent as well. * Simplify One-Time Profiler Vars * Enable restart with PSATD (ECP-WarpX#1367) * Enable restart with PSATD * Add new restart test * Add new input file * New CI Test: Fix Inputs, Fix Analysis Script * Reuse input for Restart with FDTD * Read time_of_last_gal_shift from Checkpoint * Upload Benchmark for restart_psatd * Update Benchmark for restart_psatd Co-authored-by: Edoardo Zoni <[email protected]> * Use the function `SyncRho` and the array `rho_fp` in the Electrostatic solver (ECP-WarpX#1811) * Use `SyncRho` in Electrostatic solver * Reuse rho_fp arrays * Allocate arrays for rho * Fix unused variable * Fix bug in rho deposition * Only use rho_fp in lab-frame Poisson solver * Update test cases * Incorporate PR comments * Avoid an NaN in collision module (ECP-WarpX#2225) * Update inputs_3d * Update inputs_2d * Update inputs_2d * Update inputs_3d * fix nan * removed check for final timestep diagnostics in CI tests, should revert once installafterdiagnostics is implemented Co-authored-by: Axel Huebl <[email protected]> Co-authored-by: Revathi Jambunathan <[email protected]> Co-authored-by: Edoardo Zoni <[email protected]> Co-authored-by: Neïl Zaim <[email protected]> Co-authored-by: Remi Lehe <[email protected]> Co-authored-by: Corey Wetterer-Nelson <[email protected]> Co-authored-by: Olga Shapoval <[email protected]> Co-authored-by: David Grote <[email protected]> Co-authored-by: Luca Fedeli <[email protected]> Co-authored-by: Michael Kieburtz <[email protected]> Co-authored-by: Andrew Myers <[email protected]> Co-authored-by: Peter Scherpelz <[email protected]> Co-authored-by: Edoardo Zoni <[email protected]> Co-authored-by: Yinjian Zhao <[email protected]>
ajnonaka
pushed a commit
to AMReX-Microelectronics/artemis
that referenced
this issue
Aug 26, 2021
* Rename: Optical Depths QED (#2140) This is renaming the runtime added optical depth scalars for QED physics just to create the same names in plotfile and openPMD output. * CI: Any yt works (#2066) Did not intent to commit that into mainline. * CI: Add Missing Python Analysis for EB Test (#2147) * CI: Add Missing Python Analysis for EB Test * Use 1 MPI Process for Azure * a few _rt (#2146) * PSATD: div Cleaning Implemented only with psatd.J_linear_in_time=1 (#2142) * BTD: Don't Flush If Written (#2148) Written BTD buffers for lab snapshot data are reset to zero size (count). When we do the final write of all partly filled buffers in `FilterComputePackFlushLastTimestep`, we should not write such already completed backtransformed lab snapshots again. * openPMD: `groupBased` Option Missing (#2149) The `groupBased` iteration encoding (input: `g`) was not parsed. * Remove predefined constants from example input files (#2153) * BTD_ReducedSliceDiag: BTD Plotfiles (#2152) By accident, the 2nd test did not use plotfile output. * Allow extra particle attributes (besides ux, uy, uz and w) to be set at particle creation in AddNParticles() (#2115) * exposes AddRealComp to Python to allow extra particle attributes to be added at runtime; also includes a new function to grab a particle data array from the name of the component rather than the index * added functionality to AddNParticles() to allow extra particle attributes to also be set during particle creation * added function to get index of a particle component given the PID name * changed new get component index and get_particle_arrays_from_comp_name functions to take species name as argument rather than species id * changed warpx_addRealComp to accept a species name as input and only add the new component for that species * added a test of the pywarpx bridge to get particle data and add new particle attributes at runtime * changed all particle interacting functions in libwarpx to use the species name rather than id, also changed the functions to get particle array data to use the component name rather than index * updated test according to PR #2119 changes * removed unneeded BL_ASSERT(nattr == 1) statement * fixed bug in add_particles to correctly determine the number of extra attributes * fixed bug in AddNParticles if fewer attribute values are passed than the number of extra arrays for the species * use isinstance(attr, ndarray) rather than type(attr) is np.ndarray * generalize_runtime_comps_io * fix OpenPMD * fix OpenPMD * fix plot flags in WritePlotFile * fix offset and comment * changed extra pid test to not use an underscore in the pid name * switched _libwarpx.py::add_particles to use kwargs to accept the weight and extra attribute arrays * License update in test file Co-authored-by: Axel Huebl <[email protected]> * fix typo * added a test with unique_particles=False * Apply suggestions from code review Co-authored-by: Axel Huebl <[email protected]> * updated docstring and comments Co-authored-by: atmyers <[email protected]> Co-authored-by: Axel Huebl <[email protected]> * Add predefined density profile to parameters documentation (#2155) * Release 21.08 (#2158) * AMReX/PICSAR: 21.08 * WarpX: 21.08 * initialize guard cells for macroscopic properties (#2159) * openPMD: Style Cleaning (Dec/Def) (#2160) In our code base, definitions and declarations of functions need a space after their name. This makes them easy to search for. Ref.: https://warpx.readthedocs.io/en/21.08/developers/contributing.html#style-and-conventions * openPMD: Use Steps if != BTD (#2157) * openPMD: Use Steps if != BTD For all but back-transformed diagnostics, we can use efficient, temporally sequentially increasing writes to iteration numbers for iterations. This allows us to give a guarantee to HPC I/O libraries on how to arrange data, e.g., we can use ADIOS2 BeginStep() and EndStep(). This enables paths to: - streaming workflows, such as ADIOS2 SST or SSC, where we stage data over the network instead of using files https://openpmd-api.readthedocs.io/en/0.14.0/usage/streaming.html This mitigates: - host-side memory aggregation for ADIOS2 with openPMD `groupBased` iteration encoding https://openpmd-api.readthedocs.io/en/0.14.0/backends/adios2.html#memory-usage * openPMD: Open Iterations Explicitly Explicitly open iterations. Usually, file-open operations are delayed until the first load/storeChunk operation is flush-ed. In parallel contexts where we might want to do only particle writes from a few ranks but no field wries in the future, this will avoid that we run into hangs from the non-collective nature of the first flush. The Streaming API (i.e., Series::writeIterations()) will call this method implicitly as well, but back-transformed diags (particles) might still need this. * openPMD: 0.13.0+ Needed for streaming API and collective open. * openPMD: Missing Include (#2162) Fix compile issue on Conda-Forge (Windows with Clang). * PEC Analysis: Remove Unused Imports (#2165) Fix a LGTM warning on unused imports in our PEC analysis scripts. * BinaryCollision: use more general particle data structure (#2137) * Added B field to plasma lens (#2163) * Added B field to plasma lens * Fix B field and updated CI test to include the B field * Updated benchmark * Fixed bug where specifying write_dir for particle diagnostic did not work (#2167) * Add particle weight as an explicit argument for _libwarpx.py::add_particles() (#2161) * added particle weight as an explicit argument for _libwarpx.py::add_particles() * changes requested during code review * RZ PSATD: Time Averaging for Multi-J Algorithm (#2141) * RZ PSATD: Time Averaging for Multi-J Algorithm * Fix Wrong Signs in Bm * Use Time Averaging in CI Test, Update Benchmark * Minor Fix * Shift parsing of physical/mathematical constants from hard-coding to table lookup (#2128) * Shift parsing of physical/mathematical constants from hard-coding to table lookup * Make constants map local and static for now, until there's a reason for it to be accessible/modifiable * Accept rewording * Accept rewording Co-authored-by: Axel Huebl <[email protected]> * Moved some routines to .cpp file (#2168) * In add_particles, used None for input parameter values (#2169) * More cleanup of particle boundaries (#2171) * Moved some routines to .cpp file * Moved more stuff to a header file * make parameter less prone to numerical issues in single precision (#2173) * ES Solver: Fix SP Build (#2174) Fix HIP: ``` /home/runner/work/WarpX/WarpX/Source/FieldSolver/ElectrostaticSolver.cpp:434:9: error: non-constant-expression cannot be narrowed from type 'double' to 'float' in initializer list [-Wc++11-narrowing] 1.-beta[0]*beta[0], 1.-beta[1]*beta[1], 1.-beta[2]*beta[2])}); ^~~~~~~~~~~~~~~~~~ /home/runner/work/WarpX/WarpX/build_sp/_deps/fetchedamrex-src/Src/Base/AMReX_SPACE.H:151:31: note: expanded from macro 'AMREX_D_DECL' ^ /home/runner/work/WarpX/WarpX/Source/FieldSolver/ElectrostaticSolver.cpp:434:9: note: insert an explicit cast to silence this issue 1.-beta[0]*beta[0], 1.-beta[1]*beta[1], 1.-beta[2]*beta[2])}); ^~~~~~~~~~~~~~~~~~ ``` * EB: RZ Warnings (#2176) Not an implementation yet, just adding aborts and silencing warnings. * fix typo (#2175) * Fix: Performance Tests (Boundary) (#2178) The performance tests were aborting since we changed the boundary condition inputs last month. This fixes it. * CI: Cover Performance Tests (#2179) * CI: Cover Performance Tests Make sure their syntax stays up-to-date. * Performance Test in CI: Fake Output Add "fake" output so we do not have unused variables when the tests are run. * AMReX/PICSAR: Weekly Update (#2184) Weekly update to latest AMReX. No changes in PICSAR. * Updated documentation for the plasma lens (#2182) * Updated documentation for the plasma lens * Use math formatting for B * Use math formatting for E Co-authored-by: Axel Huebl <[email protected]> * [mini-PR] Radiation Reaction : fix issue in single precision (#2186) * fix issue in single precision * Work-Around: ROCm/rocFFT <=4.3.0 (#2180) C++ template typedefs do not work in `extern C` blocks. This work-arounds this construct: https://github.com/ROCmSoftwarePlatform/rocFFT/blob/rocm-4.3.0/library/include/rocfft.h#L36-L42 This is fixed in the post 4.3.0 rocFFT `develop` branch already. Fixes build errors of the kind: ``` AnyFFT.H:15: In file included from /opt/rocm-4.1.0/hip/../include/rocfft.h:38: /usr/lib64/gcc/x86_64-suse-linux/7/../../../../include/c++/7/cstddef:69:21: error: explicit specialization of undeclared template struct '__byte_operand' template<> struct __byte_operand<bool> { using __type = byte; }; ``` * Docs: Turner Example Calibration Data (#2185) * Docs: Turner Example Calibration Data Add a note how to get the additional cross-section data for the user-facing capacitive discharge example. * Remove EOL whitespaces * Fix GetIteration (#2189) - use after free: returned a reference to a local - use copy elision to return a new iteration object * For plasma lens, mapped particle to lab frame (#2190) * For plasma lens, mapped particle to lab frame * Updated documentation for plasma lens * WarpX::Version: Not Empty (#2193) Make sure we do not return an empty string in `WarpX::Version()`. In some situations, e.g., in CI/run_test.sh, the macro for the `WARPX_GIT_VERSION` version in GNUmake is set but empty. ` Since empty versions are problematic for HDF5 attributes and confusing anyway, we return a proper `"Unknown"` in such a situation now, too. Detected as bug in #2150 X-ref: - openPMD/openPMD-api#1087 - openPMD/openPMD-api#979 * Implemented new style BTD in picmi (#2187) * Implemented new style BTD in picmi * Added rho_species. Added documentation * Docs: Fix Cori URL (404) (#2194) * Fixed issue where GPU runs would continually print 're-sorting particles' even with verbose turned off (#2197) * Docs: Summit Ascent Moved (#2195) Raider is now Twix. * AMReX/PICSAR: Weekly Update (#2199) Weekly update to latest AMReX. No changes in PICSAR. * do_pml should not be parsed anymore. (#2183) * do_pml not parsed. remove code that was added to support both types of boundary interface * add paranthesis * fix eol and move to BackwardCompatibility * missing semicolon * Update Source/WarpX.cpp * clean input files in examples * delete do_pml in performance test input * fixing an example input file in docs * Update Source/WarpX.cpp Co-authored-by: Axel Huebl <[email protected]> * cleaning up docs * Update Docs/source/usage/parameters.rst * fix eol Co-authored-by: Axel Huebl <[email protected]> * Multi-J (Hybrid): Fix Bug with Current Centering (#2181) * Use less guard cells in ParallelCopy of refined level's UpdateAuxilaryData (#2144) * Use less guard cells in ParallelCopy of refined level's UpdateAuxilaryData * Update Source/Parallelization/WarpXComm.cpp * Add inline comments and use explicit type for ng_src Co-authored-by: Edoardo Zoni <[email protected]> * Update Scripts: .ini file (#2191) Update regression tester .ini files as well in weekly/release updates of AMReX and PICSAR. * Minor fix to the documentation of the plasma lens (#2200) * CI: Cancel Prev. GH Action on Push (#2202) Save CI resources by canceling already running or waiting builds if a PR is updated. This was previously only done for Azure and now also for GH actions. * openPMD-api: 0.14.2 (#2150) This updates our requirements to openPMD-api 0.14.2+ Among others, this release introduced resizable data sets, which we will soon use for particle backtransformed diagnostics. * update use-sensei flag (#2192) * update use-sensei flag also add FlushFormatSensei.cpp to CMakeLists * CMake: WarpX SENSEI Option Co-authored-by: Axel Huebl <[email protected]> * Rho Diags: Do Not Allocate `rho_fp/cp`, Remove `plot_rho`, `setplot_rho` (#2203) * Bug fixed: looping over all species now within `OneStep_multiJ` (#2207) * Bug fixed: looping over all species now within 'OneStep_multij' * Bug fixed: looping over all laser particles within DepositCharge in multi-J * Implemented the parsing of integer input parameters (#2138) * Implemented the integer parser * Updated comment * Updated documentation * Fixed unused parameters * Added some additional documentation * Reworked the implementation so that expressions are evaluated as real and rounded to the nearest integer * Fixed loop type * Copied over initial value of variable to the real instance * Update Source/Utils/WarpXUtil.cpp make result const * Update Source/Utils/WarpXUtil.cpp make result const * Update Source/Utils/WarpXUtil.cpp Fix comment * Added safeCastToInt * Fixed adding of safeCastToInt * Cleaned up safe casting routine * Added parsing of more integer inputs * Cleaned up the integer parser, removing unneeded cast from int to real * Made x a const in safeCastToInt Co-authored-by: Neïl Zaim <[email protected]> Co-authored-by: Axel Huebl <[email protected]> * Ionization.H: Fix Bug (Division by Zero) (#2214) * Add plot_raw_rho in parameters doc (#2212) * add plot_raw_rho in parameters doc * Update Docs/source/usage/parameters.rst Co-authored-by: Luca Fedeli <[email protected]> Co-authored-by: Luca Fedeli <[email protected]> * Docs: Summit with RHEL8 (#2216) Summit has undergone a major software update to RHEL8. The default compilers and CUDA version have been modernized, among others providing C++17 support by default. Also, our scientific I/O stack is now system-provided, thanks to our Spack and E4S efforts 🎉 Please update your `warpx.profile` on Summit, re-build your Python virtual environment and re-compile your executables. * Improved error handling when the libwarpx shared object library can't be loaded (#2215) * Improved error handling when the libwarpx shared library can't be loaded * Removed extra newline * Improved check and ended program on error * Particle boundary scrape (#2124) * Some prelimary refactoring. * missing header * implement scraping particles that leave the domain boundaries into buffers * fix tabs * missing return * merging * remove redefinition * functor to work around cuda bug. * handle 2D * Add support for EB buffer * protect for AMREX_USE_EB, static_assert that EB and RZ aren't both on. * fix unused * add inputs file * add test * fix bugs, remove print * fix test * fix test path. * remove no-op code * adding clear particles method * attempt at adding time stamp * Use integer step number instead of physical time to timestamp particles; also put shared code into named functor. * move call to before apply boundary conditions * use more descriptive inputs parameter * Update Source/Particles/ParticleBoundaryBuffer.cpp * fix comp bug * move CopyAndTimestamp to cpp file * also move IsOutsideProblemDomain functor * Rename to m_particle_boundary_buffer - Name: currently only used for boundary scraping - Singular: only one instance * Fix missing EOF newline * Typo: author * Param Read / Init: Cleanup For Simplify Co-authored-by: Roelof Groenewald <[email protected]> Co-authored-by: Axel Huebl <[email protected]> * Updated the installation instructions (#2218) * Docs: OLCF ADIOS2 Currently Broken Tickets open: - OLCFHELP-3319 - ornladios/ADIOS2#2836 * Allow Silver-Mueller boundary conditions to only be applied in certain directions (#2220) * Apply Silver-Mueller boundary only in requested direction * Include a test with independent Silver-Mueller boundary conditions * Correct typo in implementation * Added transform of fields from lab to boosted frame (#2201) * Evolve: Reorder py_afterstep and cur_time break (#2213) * Docs: Cori PICMI Instructions (#2219) * Docs: Cori PICMI Instructions Too tricky to get right to just cross-link - add documentation how to build & run PICMI interfaces on Cori. * Link: Jupyter Instructions * No particle resorting when no species (#2136) * default sort interval for particles if there are not species * declare parmparse Co-authored-by: Edoardo Zoni <[email protected]> * fix eol * define and set sort_interval default to -1 and then reset them to 4 for GPU, -1 for CPU if there are species/lasers in the input Co-authored-by: Edoardo Zoni <[email protected]> * AMReX/PICSAR: Weekly Update (#2222) Weekly update to latest AMReX. No changes in PICSAR. * Scalar field interpolator from grid to particle position (#2221) * refactored distance to EB calculation to have a more general function to interpolate from a scalar field on the grid to a given position * changed RZ error message to general interpolation function * changed function names to specify that scalar field interpolation is for a nodal field only * Check for unused WarpX environment variables when compiling (#2208) * Added check for unused 'WarpX' environment variables * Changed the approach of this - Go through environment variables as they are used to set cmake flags, and then check any remaining if they start with warpx. * Improved the warning message Co-authored-by: Axel Huebl <[email protected]> * Ascent/SENSEI: Add Profiler (#2204) * Ascent/SENSEI: Add Profiler Add overall write profilers to Ascent & SENSEI. Add detailed profile for sub-operaions (blueprint, publish, execute) to Ascent as well. * Simplify One-Time Profiler Vars * Enable restart with PSATD (#1367) * Enable restart with PSATD * Add new restart test * Add new input file * New CI Test: Fix Inputs, Fix Analysis Script * Reuse input for Restart with FDTD * Read time_of_last_gal_shift from Checkpoint * Upload Benchmark for restart_psatd * Update Benchmark for restart_psatd Co-authored-by: Edoardo Zoni <[email protected]> * Use the function `SyncRho` and the array `rho_fp` in the Electrostatic solver (#1811) * Use `SyncRho` in Electrostatic solver * Reuse rho_fp arrays * Allocate arrays for rho * Fix unused variable * Fix bug in rho deposition * Only use rho_fp in lab-frame Poisson solver * Update test cases * Incorporate PR comments * Avoid an NaN in collision module (#2225) * Update inputs_3d * Update inputs_2d * Update inputs_2d * Update inputs_3d * fix nan * Docs: Fix Summit Python (#2227) We cannot use an anaconda base package, since anaconda ships its own libstdc++ for its compiler stack, which is not compatible with our compiler choice. * WarpXComm.cpp: Fix Bug (Uninitialized MultiFabs) (#2230) * Alphabetized and added PR to glossary (#2228) * Fix: Missing Include Guards (FWD) (#2226) * Fix: Missing Include Guards (FWD) All header files need include guards to avoid double definitons. * Fix copy-paste guard * Avoid Unnecessary FFTs of rho (#2234) * rearranging the loops (#2231) * GetPosition 2D3V: y=0 (#2233) In 2D3V, we currently assigned the y position a signaling NaN in `GetPosition`, so that it cannot be used accidentially in functions. When we call parser functions, this value is copied even if it is not used, raising the signaling NaN. Rethinking the strategy, we can write more general inputs files if we instead write a sensible default for `y` in 2D3V: `y=0`. So this PR changes the default, which also fixes the issue. * Multi-J: Optimize Allocations of rho MultiFabs (#2224) * OneStep_multiJ: Always Deposit rho in Component 0 * For Multi-J Algo Allocate only 1 Component Co-authored-by: Remi Lehe <[email protected]> * Fix SENSEI & Ascent component in AMReX.cmake (#2235) * fix sensei component name in AMReX.cmake * CMake: Fix Ascent/Conduit Component Co-authored-by: Axel Huebl <[email protected]> * Update AMReX: Parser, SENSEI (#2236) Update AMReX to pull in new changes about: - local variables in the parser are not reported as undefined - SENSEI fixes on AMRLevel (now truly opt-in) Co-authored-by: Axel Huebl <[email protected]> Co-authored-by: Edoardo Zoni <[email protected]> Co-authored-by: MaxThevenet <[email protected]> Co-authored-by: Neïl Zaim <[email protected]> Co-authored-by: Roelof Groenewald <[email protected]> Co-authored-by: atmyers <[email protected]> Co-authored-by: David Grote <[email protected]> Co-authored-by: Kevin Z. Zhu <[email protected]> Co-authored-by: Phil Miller <[email protected]> Co-authored-by: Luca Fedeli <[email protected]> Co-authored-by: Remi Lehe <[email protected]> Co-authored-by: Corey Wetterer-Nelson <[email protected]> Co-authored-by: Olga Shapoval <[email protected]> Co-authored-by: Luca Fedeli <[email protected]> Co-authored-by: Michael Kieburtz <[email protected]> Co-authored-by: Andrew Myers <[email protected]> Co-authored-by: Peter Scherpelz <[email protected]> Co-authored-by: Edoardo Zoni <[email protected]> Co-authored-by: Yinjian Zhao <[email protected]> Co-authored-by: Tiberius Rheaume <[email protected]> Co-authored-by: Lorenzo Giacomel <[email protected]>
|
@mpbelhorn thanks a lot, I am doing an integrated test now and will report in the ticket. |
Correct - in theory, it should not be necessary to load any |
|
Thanks a lot, that's fantastic! Reported also in the ticket: Thank you all for the great team work to get this resolved so quickly ❤️ |
|
cc @franzpoeschel: when you are back from vacation, we could repeat the pre-RHEL8 SST tests that you did with the new Summit module for ADIOS2. Maybe that new libfabric core helps with those instabilities/crashes at large node numbers that we saw prior to the system update. Otherwise, let us follow up with SST reproducers in a new ticket. |
This was referenced Aug 26, 2021
|
Just added to the OLCF ticket: when going to a few 100 nodes, we see new errors of the kind with the ADIOS module on Summit. The problem does not show up at smaller scale. |
|
Opened a new ticket in OLCFHELP-3545 since the old one is closed. |
mpbelhorn
added a commit
to mpbelhorn/olcf-spack-environments
that referenced
this issue
Sep 9, 2021
Snapshots environment prior to reconcretizing environment for system RDMA-core external package needed for libfabric to address issues with Adios2: ornladios/ADIOS2#2836 https://jira.ccs.ornl.gov/projects/OLCFHELP/queues/custom/44/OLCFHELP-3319
ajnonaka
pushed a commit
to AMReX-Microelectronics/artemis
that referenced
this issue
Oct 4, 2021
* Rename: Optical Depths QED (#2140) This is renaming the runtime added optical depth scalars for QED physics just to create the same names in plotfile and openPMD output. * CI: Any yt works (#2066) Did not intent to commit that into mainline. * CI: Add Missing Python Analysis for EB Test (#2147) * CI: Add Missing Python Analysis for EB Test * Use 1 MPI Process for Azure * a few _rt (#2146) * PSATD: div Cleaning Implemented only with psatd.J_linear_in_time=1 (#2142) * BTD: Don't Flush If Written (#2148) Written BTD buffers for lab snapshot data are reset to zero size (count). When we do the final write of all partly filled buffers in `FilterComputePackFlushLastTimestep`, we should not write such already completed backtransformed lab snapshots again. * openPMD: `groupBased` Option Missing (#2149) The `groupBased` iteration encoding (input: `g`) was not parsed. * Remove predefined constants from example input files (#2153) * BTD_ReducedSliceDiag: BTD Plotfiles (#2152) By accident, the 2nd test did not use plotfile output. * Allow extra particle attributes (besides ux, uy, uz and w) to be set at particle creation in AddNParticles() (#2115) * exposes AddRealComp to Python to allow extra particle attributes to be added at runtime; also includes a new function to grab a particle data array from the name of the component rather than the index * added functionality to AddNParticles() to allow extra particle attributes to also be set during particle creation * added function to get index of a particle component given the PID name * changed new get component index and get_particle_arrays_from_comp_name functions to take species name as argument rather than species id * changed warpx_addRealComp to accept a species name as input and only add the new component for that species * added a test of the pywarpx bridge to get particle data and add new particle attributes at runtime * changed all particle interacting functions in libwarpx to use the species name rather than id, also changed the functions to get particle array data to use the component name rather than index * updated test according to PR #2119 changes * removed unneeded BL_ASSERT(nattr == 1) statement * fixed bug in add_particles to correctly determine the number of extra attributes * fixed bug in AddNParticles if fewer attribute values are passed than the number of extra arrays for the species * use isinstance(attr, ndarray) rather than type(attr) is np.ndarray * generalize_runtime_comps_io * fix OpenPMD * fix OpenPMD * fix plot flags in WritePlotFile * fix offset and comment * changed extra pid test to not use an underscore in the pid name * switched _libwarpx.py::add_particles to use kwargs to accept the weight and extra attribute arrays * License update in test file Co-authored-by: Axel Huebl <[email protected]> * fix typo * added a test with unique_particles=False * Apply suggestions from code review Co-authored-by: Axel Huebl <[email protected]> * updated docstring and comments Co-authored-by: atmyers <[email protected]> Co-authored-by: Axel Huebl <[email protected]> * Add predefined density profile to parameters documentation (#2155) * Release 21.08 (#2158) * AMReX/PICSAR: 21.08 * WarpX: 21.08 * initialize guard cells for macroscopic properties (#2159) * openPMD: Style Cleaning (Dec/Def) (#2160) In our code base, definitions and declarations of functions need a space after their name. This makes them easy to search for. Ref.: https://warpx.readthedocs.io/en/21.08/developers/contributing.html#style-and-conventions * openPMD: Use Steps if != BTD (#2157) * openPMD: Use Steps if != BTD For all but back-transformed diagnostics, we can use efficient, temporally sequentially increasing writes to iteration numbers for iterations. This allows us to give a guarantee to HPC I/O libraries on how to arrange data, e.g., we can use ADIOS2 BeginStep() and EndStep(). This enables paths to: - streaming workflows, such as ADIOS2 SST or SSC, where we stage data over the network instead of using files https://openpmd-api.readthedocs.io/en/0.14.0/usage/streaming.html This mitigates: - host-side memory aggregation for ADIOS2 with openPMD `groupBased` iteration encoding https://openpmd-api.readthedocs.io/en/0.14.0/backends/adios2.html#memory-usage * openPMD: Open Iterations Explicitly Explicitly open iterations. Usually, file-open operations are delayed until the first load/storeChunk operation is flush-ed. In parallel contexts where we might want to do only particle writes from a few ranks but no field wries in the future, this will avoid that we run into hangs from the non-collective nature of the first flush. The Streaming API (i.e., Series::writeIterations()) will call this method implicitly as well, but back-transformed diags (particles) might still need this. * openPMD: 0.13.0+ Needed for streaming API and collective open. * openPMD: Missing Include (#2162) Fix compile issue on Conda-Forge (Windows with Clang). * PEC Analysis: Remove Unused Imports (#2165) Fix a LGTM warning on unused imports in our PEC analysis scripts. * BinaryCollision: use more general particle data structure (#2137) * Added B field to plasma lens (#2163) * Added B field to plasma lens * Fix B field and updated CI test to include the B field * Updated benchmark * Fixed bug where specifying write_dir for particle diagnostic did not work (#2167) * Add particle weight as an explicit argument for _libwarpx.py::add_particles() (#2161) * added particle weight as an explicit argument for _libwarpx.py::add_particles() * changes requested during code review * RZ PSATD: Time Averaging for Multi-J Algorithm (#2141) * RZ PSATD: Time Averaging for Multi-J Algorithm * Fix Wrong Signs in Bm * Use Time Averaging in CI Test, Update Benchmark * Minor Fix * Shift parsing of physical/mathematical constants from hard-coding to table lookup (#2128) * Shift parsing of physical/mathematical constants from hard-coding to table lookup * Make constants map local and static for now, until there's a reason for it to be accessible/modifiable * Accept rewording * Accept rewording Co-authored-by: Axel Huebl <[email protected]> * Moved some routines to .cpp file (#2168) * In add_particles, used None for input parameter values (#2169) * More cleanup of particle boundaries (#2171) * Moved some routines to .cpp file * Moved more stuff to a header file * make parameter less prone to numerical issues in single precision (#2173) * ES Solver: Fix SP Build (#2174) Fix HIP: ``` /home/runner/work/WarpX/WarpX/Source/FieldSolver/ElectrostaticSolver.cpp:434:9: error: non-constant-expression cannot be narrowed from type 'double' to 'float' in initializer list [-Wc++11-narrowing] 1.-beta[0]*beta[0], 1.-beta[1]*beta[1], 1.-beta[2]*beta[2])}); ^~~~~~~~~~~~~~~~~~ /home/runner/work/WarpX/WarpX/build_sp/_deps/fetchedamrex-src/Src/Base/AMReX_SPACE.H:151:31: note: expanded from macro 'AMREX_D_DECL' ^ /home/runner/work/WarpX/WarpX/Source/FieldSolver/ElectrostaticSolver.cpp:434:9: note: insert an explicit cast to silence this issue 1.-beta[0]*beta[0], 1.-beta[1]*beta[1], 1.-beta[2]*beta[2])}); ^~~~~~~~~~~~~~~~~~ ``` * EB: RZ Warnings (#2176) Not an implementation yet, just adding aborts and silencing warnings. * fix typo (#2175) * Fix: Performance Tests (Boundary) (#2178) The performance tests were aborting since we changed the boundary condition inputs last month. This fixes it. * CI: Cover Performance Tests (#2179) * CI: Cover Performance Tests Make sure their syntax stays up-to-date. * Performance Test in CI: Fake Output Add "fake" output so we do not have unused variables when the tests are run. * AMReX/PICSAR: Weekly Update (#2184) Weekly update to latest AMReX. No changes in PICSAR. * Updated documentation for the plasma lens (#2182) * Updated documentation for the plasma lens * Use math formatting for B * Use math formatting for E Co-authored-by: Axel Huebl <[email protected]> * [mini-PR] Radiation Reaction : fix issue in single precision (#2186) * fix issue in single precision * Work-Around: ROCm/rocFFT <=4.3.0 (#2180) C++ template typedefs do not work in `extern C` blocks. This work-arounds this construct: https://github.com/ROCmSoftwarePlatform/rocFFT/blob/rocm-4.3.0/library/include/rocfft.h#L36-L42 This is fixed in the post 4.3.0 rocFFT `develop` branch already. Fixes build errors of the kind: ``` AnyFFT.H:15: In file included from /opt/rocm-4.1.0/hip/../include/rocfft.h:38: /usr/lib64/gcc/x86_64-suse-linux/7/../../../../include/c++/7/cstddef:69:21: error: explicit specialization of undeclared template struct '__byte_operand' template<> struct __byte_operand<bool> { using __type = byte; }; ``` * Docs: Turner Example Calibration Data (#2185) * Docs: Turner Example Calibration Data Add a note how to get the additional cross-section data for the user-facing capacitive discharge example. * Remove EOL whitespaces * Fix GetIteration (#2189) - use after free: returned a reference to a local - use copy elision to return a new iteration object * For plasma lens, mapped particle to lab frame (#2190) * For plasma lens, mapped particle to lab frame * Updated documentation for plasma lens * WarpX::Version: Not Empty (#2193) Make sure we do not return an empty string in `WarpX::Version()`. In some situations, e.g., in CI/run_test.sh, the macro for the `WARPX_GIT_VERSION` version in GNUmake is set but empty. ` Since empty versions are problematic for HDF5 attributes and confusing anyway, we return a proper `"Unknown"` in such a situation now, too. Detected as bug in #2150 X-ref: - https://github.com/openPMD/openPMD-api/pull/1087 - https://github.com/openPMD/openPMD-api/pull/979 * Implemented new style BTD in picmi (#2187) * Implemented new style BTD in picmi * Added rho_species. Added documentation * Docs: Fix Cori URL (404) (#2194) * Fixed issue where GPU runs would continually print 're-sorting particles' even with verbose turned off (#2197) * Docs: Summit Ascent Moved (#2195) Raider is now Twix. * AMReX/PICSAR: Weekly Update (#2199) Weekly update to latest AMReX. No changes in PICSAR. * do_pml should not be parsed anymore. (#2183) * do_pml not parsed. remove code that was added to support both types of boundary interface * add paranthesis * fix eol and move to BackwardCompatibility * missing semicolon * Update Source/WarpX.cpp * clean input files in examples * delete do_pml in performance test input * fixing an example input file in docs * Update Source/WarpX.cpp Co-authored-by: Axel Huebl <[email protected]> * cleaning up docs * Update Docs/source/usage/parameters.rst * fix eol Co-authored-by: Axel Huebl <[email protected]> * Multi-J (Hybrid): Fix Bug with Current Centering (#2181) * Use less guard cells in ParallelCopy of refined level's UpdateAuxilaryData (#2144) * Use less guard cells in ParallelCopy of refined level's UpdateAuxilaryData * Update Source/Parallelization/WarpXComm.cpp * Add inline comments and use explicit type for ng_src Co-authored-by: Edoardo Zoni <[email protected]> * Update Scripts: .ini file (#2191) Update regression tester .ini files as well in weekly/release updates of AMReX and PICSAR. * Minor fix to the documentation of the plasma lens (#2200) * CI: Cancel Prev. GH Action on Push (#2202) Save CI resources by canceling already running or waiting builds if a PR is updated. This was previously only done for Azure and now also for GH actions. * openPMD-api: 0.14.2 (#2150) This updates our requirements to openPMD-api 0.14.2+ Among others, this release introduced resizable data sets, which we will soon use for particle backtransformed diagnostics. * update use-sensei flag (#2192) * update use-sensei flag also add FlushFormatSensei.cpp to CMakeLists * CMake: WarpX SENSEI Option Co-authored-by: Axel Huebl <[email protected]> * Rho Diags: Do Not Allocate `rho_fp/cp`, Remove `plot_rho`, `setplot_rho` (#2203) * Bug fixed: looping over all species now within `OneStep_multiJ` (#2207) * Bug fixed: looping over all species now within 'OneStep_multij' * Bug fixed: looping over all laser particles within DepositCharge in multi-J * Implemented the parsing of integer input parameters (#2138) * Implemented the integer parser * Updated comment * Updated documentation * Fixed unused parameters * Added some additional documentation * Reworked the implementation so that expressions are evaluated as real and rounded to the nearest integer * Fixed loop type * Copied over initial value of variable to the real instance * Update Source/Utils/WarpXUtil.cpp make result const * Update Source/Utils/WarpXUtil.cpp make result const * Update Source/Utils/WarpXUtil.cpp Fix comment * Added safeCastToInt * Fixed adding of safeCastToInt * Cleaned up safe casting routine * Added parsing of more integer inputs * Cleaned up the integer parser, removing unneeded cast from int to real * Made x a const in safeCastToInt Co-authored-by: Neïl Zaim <[email protected]> Co-authored-by: Axel Huebl <[email protected]> * Ionization.H: Fix Bug (Division by Zero) (#2214) * Add plot_raw_rho in parameters doc (#2212) * add plot_raw_rho in parameters doc * Update Docs/source/usage/parameters.rst Co-authored-by: Luca Fedeli <[email protected]> Co-authored-by: Luca Fedeli <[email protected]> * Docs: Summit with RHEL8 (#2216) Summit has undergone a major software update to RHEL8. The default compilers and CUDA version have been modernized, among others providing C++17 support by default. Also, our scientific I/O stack is now system-provided, thanks to our Spack and E4S efforts :tada: Please update your `warpx.profile` on Summit, re-build your Python virtual environment and re-compile your executables. * Improved error handling when the libwarpx shared object library can't be loaded (#2215) * Improved error handling when the libwarpx shared library can't be loaded * Removed extra newline * Improved check and ended program on error * Particle boundary scrape (#2124) * Some prelimary refactoring. * missing header * implement scraping particles that leave the domain boundaries into buffers * fix tabs * missing return * merging * remove redefinition * functor to work around cuda bug. * handle 2D * Add support for EB buffer * protect for AMREX_USE_EB, static_assert that EB and RZ aren't both on. * fix unused * add inputs file * add test * fix bugs, remove print * fix test * fix test path. * remove no-op code * adding clear particles method * attempt at adding time stamp * Use integer step number instead of physical time to timestamp particles; also put shared code into named functor. * move call to before apply boundary conditions * use more descriptive inputs parameter * Update Source/Particles/ParticleBoundaryBuffer.cpp * fix comp bug * move CopyAndTimestamp to cpp file * also move IsOutsideProblemDomain functor * Rename to m_particle_boundary_buffer - Name: currently only used for boundary scraping - Singular: only one instance * Fix missing EOF newline * Typo: author * Param Read / Init: Cleanup For Simplify Co-authored-by: Roelof Groenewald <[email protected]> Co-authored-by: Axel Huebl <[email protected]> * Updated the installation instructions (#2218) * Docs: OLCF ADIOS2 Currently Broken Tickets open: - OLCFHELP-3319 - https://github.com/ornladios/ADIOS2/issues/2836 * Allow Silver-Mueller boundary conditions to only be applied in certain directions (#2220) * Apply Silver-Mueller boundary only in requested direction * Include a test with independent Silver-Mueller boundary conditions * Correct typo in implementation * Added transform of fields from lab to boosted frame (#2201) * Evolve: Reorder py_afterstep and cur_time break (#2213) * Docs: Cori PICMI Instructions (#2219) * Docs: Cori PICMI Instructions Too tricky to get right to just cross-link - add documentation how to build & run PICMI interfaces on Cori. * Link: Jupyter Instructions * No particle resorting when no species (#2136) * default sort interval for particles if there are not species * declare parmparse Co-authored-by: Edoardo Zoni <[email protected]> * fix eol * define and set sort_interval default to -1 and then reset them to 4 for GPU, -1 for CPU if there are species/lasers in the input Co-authored-by: Edoardo Zoni <[email protected]> * AMReX/PICSAR: Weekly Update (#2222) Weekly update to latest AMReX. No changes in PICSAR. * Scalar field interpolator from grid to particle position (#2221) * refactored distance to EB calculation to have a more general function to interpolate from a scalar field on the grid to a given position * changed RZ error message to general interpolation function * changed function names to specify that scalar field interpolation is for a nodal field only * Check for unused WarpX environment variables when compiling (#2208) * Added check for unused 'WarpX' environment variables * Changed the approach of this - Go through environment variables as they are used to set cmake flags, and then check any remaining if they start with warpx. * Improved the warning message Co-authored-by: Axel Huebl <[email protected]> * Ascent/SENSEI: Add Profiler (#2204) * Ascent/SENSEI: Add Profiler Add overall write profilers to Ascent & SENSEI. Add detailed profile for sub-operaions (blueprint, publish, execute) to Ascent as well. * Simplify One-Time Profiler Vars * Enable restart with PSATD (#1367) * Enable restart with PSATD * Add new restart test * Add new input file * New CI Test: Fix Inputs, Fix Analysis Script * Reuse input for Restart with FDTD * Read time_of_last_gal_shift from Checkpoint * Upload Benchmark for restart_psatd * Update Benchmark for restart_psatd Co-authored-by: Edoardo Zoni <[email protected]> * Use the function `SyncRho` and the array `rho_fp` in the Electrostatic solver (#1811) * Use `SyncRho` in Electrostatic solver * Reuse rho_fp arrays * Allocate arrays for rho * Fix unused variable * Fix bug in rho deposition * Only use rho_fp in lab-frame Poisson solver * Update test cases * Incorporate PR comments * Avoid an NaN in collision module (#2225) * Update inputs_3d * Update inputs_2d * Update inputs_2d * Update inputs_3d * fix nan * Docs: Fix Summit Python (#2227) We cannot use an anaconda base package, since anaconda ships its own libstdc++ for its compiler stack, which is not compatible with our compiler choice. * WarpXComm.cpp: Fix Bug (Uninitialized MultiFabs) (#2230) * Alphabetized and added PR to glossary (#2228) * Fix: Missing Include Guards (FWD) (#2226) * Fix: Missing Include Guards (FWD) All header files need include guards to avoid double definitons. * Fix copy-paste guard * Avoid Unnecessary FFTs of rho (#2234) * rearranging the loops (#2231) * GetPosition 2D3V: y=0 (#2233) In 2D3V, we currently assigned the y position a signaling NaN in `GetPosition`, so that it cannot be used accidentially in functions. When we call parser functions, this value is copied even if it is not used, raising the signaling NaN. Rethinking the strategy, we can write more general inputs files if we instead write a sensible default for `y` in 2D3V: `y=0`. So this PR changes the default, which also fixes the issue. * Multi-J: Optimize Allocations of rho MultiFabs (#2224) * OneStep_multiJ: Always Deposit rho in Component 0 * For Multi-J Algo Allocate only 1 Component Co-authored-by: Remi Lehe <[email protected]> * Fix SENSEI & Ascent component in AMReX.cmake (#2235) * fix sensei component name in AMReX.cmake * CMake: Fix Ascent/Conduit Component Co-authored-by: Axel Huebl <[email protected]> * Update AMReX: Parser, SENSEI (#2236) Update AMReX to pull in new changes about: - local variables in the parser are not reported as undefined - SENSEI fixes on AMRLevel (now truly opt-in) * add include to fix compiler error (#2237) * Particle Unit Conversion: Generalize (#2223) Generalize the member to convert the internal proper velocity to SI momentum and back. This is needed for BTD diagnostics, where we have a less specialized but pinned AMReX particle container, which does not have this method. * Laser-Ion Example: Fix Histogram Functs (#2232) Avoid `sqrt(0)` and `1/sqrt(0)` operations in the inputs for the 2D3V Laser-Ion Acceleration example. Those can raise floating point exceptions and NaN's for resting particles. * Improvement in electrostic field solve when using EB & a small refactoring of the Electrostatic solver (#2143) * refactored parts of the electrostatic solver to allow an outside Poisson solver to be installed * added callback to install a Poisson solver that can be used instead of the MLMG solver * added call to the poissonsolver callback if one is installed and updated ElectrostaticSolver.cpp to calculate the electric field directly in amrex if EBs are used * fixed issue causing ElectrostaticSphereEB test to crash * fixed the logic for when to call computeE * added function to allow charge density to be deposited in rho_fp from outside WarpX * fixed inconsistency between labframe and relativistic electrostatic simulation setup * fixed typo * fixed issue that caused ElectrostaticSphere test to fail and removed unnecessary call to multiply E_y by -1 when doing a 2d simulation * revert edit to error handling string * changes requested during PR review * additional change requested during PR review * fixed typo * revert unnecessary changes * Use ROCm 4.3 in CI (#2240) Since the VOP bug has been fixed in the latest ROCm release v4.3, CI can use the latest ROCm release now instead of 4.1. We also need to work around an Ubuntu 20.04 LTS / OpenMPI 4.0.3 bug with Clang compiles regarding the missing linker flag for libopen-pal Co-authored-by: Weiqun Zhang <[email protected]> * Functionality to save particle positions from the previous step (#2206) * added functionality to save particle positions from the previous step * copied WarpX variable to local variable to fix issue with GPU and DPC++ compilation * switched to using a species attribute to toggle whether previous positions are saved so it can be turned on for only a subset of species if desired * changed variable name to be more verbose * added CI test of saving the previous particle positions * start of a table in the documentation to describe commonly used runtime attributes * generate test benchmark data from results obtained with a 2 processor run - the same as what is done during the test * relaxed tolerance on test * regenerate CI test reference data with USE_PSATD=TRUE * Update Docs/source/developers/particles.rst Co-authored-by: Phil Miller <[email protected]> Co-authored-by: Axel Huebl <[email protected]> * Ascent: Include for Profiler (#2238) Forgot an include. * Fix EB: CUDA Lambda Visibility (#2247) ``` error #3196-D: The enclosing parent function ("ComputeEdgeLengths") for an extended __device__ lambda cannot have private or protected access within its class ``` For: - `ComputeEdgeLengths` - `ComputeFaceAreas` - `ScaleEdges` - `ScaleAreas` * CI: Trap NaNs, Divisions by Zero, Overflows (#2205) * ECT conformal solver (#1923) * adding base implementation of the conformal solver * adding some preprocessor directives * qualifying the isnan's correctly * getting rid of some unused fields * computing S_stab on the fly * using empty in length check * Source/FieldSolver/FiniteDifferenceSolver/EvolveB.cpp * replaced a looponcpu with a parallelfor * trying to not pass lending/borrowing info by reference in evolveB * passing data using dataPtr * converting inds into a device vector * simplifying some += * extracting the inds BaseFabs as DataPtr * Revert "extracting the inds BaseFabs as DataPtr" This reverts commit bc709d2fcaa347c119514de79a3f57169a05af65. * implementing new way ogf handling cell extensions (gpu compatible) * fixing some white spaces * removed a typo * made a function static * tidying up * tidying up * getting rid of the rho vector * moving the rho update to evolveE * refectoring the cell extension * small chenge in input parameters * updating WarpX.H * bug fix * fixing another bug * rotating the cube * updated gitignore * using the conformal soler in the cube test * fixing the signature of a function * trying to fix the setVal problem * Revert "trying to fix the setVal problem" This reverts commit c7d0e5e3709b730ff570183b2a6df5f587ca4640. * trying to fix the setVal problem * cleaning some comments * removing an abort in device code * Including geometric information in the external field initializer * cleaned up the test * moving the geometric data initialization before the external fields initialization * changing way of saving info about intruded cells * fixed some white spaces * Using S_mod instead of S_red and S_enl * substituting a std::vector with amrex::Array1D * bug fix in the uint8_t -> coords maps * Condensed ect cell info into one single MultiFab * fixing some includes * fixing some more includes * fixed a typo * making some functions available in gpu code * using tilebox instead of validbox for the geometry initialization * communicating the geometric info in the guard cells * fixing the guard cells initialization for ect * fixing an unused parameter * fixing an unused parameter * ignored some unused variables when EB is not supported * cleaning up * cleaning up * ignored some unused variables when EB is not supported * evolving rho just for ect * handling some more unused variables * removing some white spaces * adding the rotated cube test * removed some white spaces * change cells into faces * small fix in unused parameters * fixed a comment * for safety for now just initialize the geometric data when lev==maxLevel * adding some preprocessor directives to isolate EB code * updating Makefile stuff * Improving the rotated cube test * bug fix in mesh refinement * fixing boundary conditions in rotated cube test * deactivating MPI in rotated cube test * activating mpi in test * improving outputs of cells extension * improved the documentation * Update Source/EmbeddedBoundary/WarpXInitEB.cpp Co-authored-by: Weiqun Zhang <[email protected]> * Update Source/EmbeddedBoundary/WarpXInitEB.cpp Co-authored-by: Weiqun Zhang <[email protected]> * Removed some obsolete isnan's * undone change to gitignore * adding some missing AMREX_GPU_DEVICEs * Changing some amrex::Real(0) into 0. Co-authored-by: Edoardo Zoni <[email protected]> * Making some variables const Co-authored-by: Edoardo Zoni <[email protected]> * Adding amrex:: to some ignore_unused Co-authored-by: Edoardo Zoni <[email protected]> * Enhancing readability of some parallelfor's Co-authored-by: Edoardo Zoni <[email protected]> * Removed some unecessary includes * Fixing some tags Co-authored-by: Edoardo Zoni <[email protected]> * Fixing the function CountExtFaces * added a comment for Rhofield * Fixed a typo in CountExtCells * trying to remove accesses to private members in ComputeEdgeLengths * making some functions public * undoing some useless changes * undoing some useless changes * adding some AMREX_GPU_DEVICEs * adding some unused variables * adding some AMREX_GPU_DEVICEs to fix warnings * fixing relative error * Making several variables const Co-authored-by: Edoardo Zoni <[email protected]> * Fixed a comment Co-authored-by: Edoardo Zoni <[email protected]> * changing some double to amrex::Real * removing commented lines * changing some double to amrex::Real * removing commented lines * inizialing nelems_x,y,z to avoid a warning * Fixing number of cells * Adding missing analysis routine in rotated cube test Co-authored-by: Edoardo Zoni <[email protected]> * Making some variables constexpr Co-authored-by: Weiqun Zhang <[email protected]> * Improving some reduce operations Co-authored-by: Weiqun Zhang <[email protected]> * reading time directly from the output * fixed a few data types * fixing another int * Replacing or->||, and->&&, not->! * Adding license info * Adding a white space Co-authored-by: Axel Huebl <[email protected]> * removed some unused imports * Moving the doxygen documentation * Including AMReX_LayoutData.H * removing some useless parameters * adding ect solver to the doc * Removing some useless reductions * Small change for consistency * Handling the resizing of arrays * Handling correctly the resing of arrays * Fixing some whitespaces * Fixing an indentation * Improving a comment * Removed a useless initialization * Renamed Rhofield to ECTRhofield * Renamed Rhofield to ECTRhofield * Added some if conditions to isolate some EB related code * Improved some comments Co-authored-by: Remi Lehe <[email protected]> * Made some functions not member of WarpX anymore * Isolated some EB-only code * Isolated some EB-only code * Using the square brackets to access vectors * Ignoring some unused variables * Bug fix in initialization * Removed a redundant if * Using enumeration for connectivity and improved names * Added a comment * added a few comments * Removed some useless template parameters * Printing some info only in verbose mode * Revert "Removed some useless template parameters" This reverts commit 2c527980e6872c0212767c27f00e2b53ecbcfd0a. * Fixed a bug with the neighbours enumeration * Initializing geom data alsoo in the ghost cells Co-authored-by: Weiqun Zhang <[email protected]> Co-authored-by: Edoardo Zoni <[email protected]> Co-authored-by: Axel Huebl <[email protected]> Co-authored-by: Remi Lehe <[email protected]> * CI: Rename Travis -> CI (#2250) Rename scripts and docs since our CI uses several providers now (currently: GitHub actions, Azure pipelines and other providers). * AMReX/PICSAR: Weekly Update (#2249) Weekly update to latest AMReX. No changes in PICSAR. * Make buffer of scraped particles available to Python code (#2164) * Added wrapper to get number of particle species tracked by the scraper Not sure if this is going to be useful, but it demonstrates a method to get information from the ParticleBoundaryBuffer into Python. * Stubbed out the main wrapper functions * Added parameters to wrapper * Added wrapper for getting the number of particles scraped of a species on a boundary * added picmi arguments to scrape particles at the domain boundary * Added wrapper to get the full particle buffer into python * rearanged the getBuffer properties code a little * Added docstrings +other suggested changes * Added num_particles_impacted_boundary docstring * fixed mistake in docstring * Changed boundary parameter to be a string for clarity * Fixed issue with the boundary parameter for scraping * Fixed issue with the boundary input for scraping stats wrapper * Added demonstration of particle scraping wrapper * Added analysis.py file * Fix typo in one of the dimension maps Co-authored-by: Roelof Groenewald <[email protected]> * Added before esolve to warpx evolve * added test for the scraped particle buffer wrappers * Moved python PICMI particle boundary scrape test * Renamed test file to the correct name * Removed old test * added special functionality to get the timestep at which particles were scraped * removed debug print * added python wrapper for the clearParticles() function of the scraper buffer * added special wrapper function to get the timesteps at which the particles in the boundary buffer were scraped * updated test to match the non-PICMI test for the particle scraper buffer * Fix uncaught rebase mistake * re-activated picmi test of accessing the scraped particle buffers via python * added documentation for the new parameters involved in the scraped particle buffer and fixed remaining issue with picmi test * changes requested during code review Co-authored-by: mkieburtz <[email protected]> Co-authored-by: Roelof <[email protected]> Co-authored-by: Roelof Groenewald <[email protected]> * Docs: Perlmutter (#2229) * Docs: Perlmutter Start a documentation page for Perlmutter. * Cleaning - better links to docs - clean submission script * Perlmutter: Add I/O * Docs: ADIOS2 Fixed on Summit (#2239) The adios2 system module is now fixed on Summit. We don't need to load the openpmd-api module for CMake, as we build it on-the-fly at the moment against the ADIOS2 and HDF5 modules. * Glossary: OTP/2FA (#2251) Abbreviation used for NERSC/HPC users * Added parsing in intervals (#2246) * Added parsing in intervals, minor clean up of querywithparser routines * Split up documentation * Added documentation * Added tests to inputs_2d_multi_rz_rt Co-authored-by: Axel Huebl <[email protected]> * Add profiling for charge deposition (#2253) * Add link to repo visual structure (#2256) * Add link to repo visual structure Add a link to the GitHub visualization of the repository structure. Full description at https://next.github.com/projects/repo-visualization * Link: md->rst syntax * Initializing geometric data also with the CKC solver (#2254) * Docs: Summit umask (Permissions) (#2260) After the update to RHEL8, Summit changed the default permissions of newly created files and directories (for the worse). This has been reported as OLCFHELP-3442 but we face some resistance from support to triage this properly at the moment. Since we need to continue to keep working, we change the `umask` (aka defaults for new files & dirs) manually so that group members of the same project can read files and access dirs. For files and dirs created since this update and not yet using this `umask`, please use the following fix. ``` find . -type -d -exec chmod a+rx {} \; find . -type -f -exec chmod a+r {} \; ``` Replace `.` (current directory) with another path if needed. * Docs: Perlmutter Aug 30th Update (#2262) Module changes from: https://docs.nersc.gov/systems/perlmutter/timeline/#august-30-2021 * [mini-PR] Improved doc on fwd headers (#2267) * improved doc on fwd headers * fix bug * fix bug * silence unused params errors in FlushFormatSensei (#2265) * Bug fix in call to RhoFunctor for RZ modes (#2271) * Release 21.09 (#2270) * Update: AMReX/PICSAR to 21.09 * WarpX: 21.09 * Initializing the size BaseFab on Device (#2255) * Docs: Summit w/o Darshan (#2272) We see problems again at runtime with missing libs. We don't need Darshan by default, so let's unload it to make our stack more stable. We also avoid sourcing the profile again in the batch script: we inherit the environment and modules loaded at submission. * Docs: Summit Ascent Update (#2241) * Docs: Summit Ascent Update Update the Summit Ascent install location. * new dir for gcc 9.3.0 * Update: HDF5 fix * AMReX Update: SENSEI (#2276) Update AMReX to include: - https://github.com/AMReX-Codes/amrex/pull/2283 - https://github.com/AMReX-Codes/amrex/pull/2285 * CI: Build EB (#2170) * CI: Build EB Perform compile tests with EB on. This should help to ensure CUDA, DPC++, HIP and Windows builds work with EB on the long-term. * CI Wheels: EB + RZ not yet supported * DPC++: Compiler ICE with EB Disable for now and report upstream in a follow-up. * Summit: PICMI Update (#2275) * Summit: PICMI Update Update PICMI instructions post the RHEL8 upgrade of Summit (OLCF). * Summit: Batch-Node Compilation * Add optional ums modules * Summit: Work-Around IBM MPI Collectives (#2283) Fix crashes on Summit at scale (>~224 nodes) due to failing Barriers in IBM's MPI stack. Seen mostly with I/O routines and only since the RHEL8 upgrade. We see no signfiicant performance impact from this work-around until OLCF & IBM fix the problem (OLCFHELP-3545). An alternative work-around via ``` export OMPI_MCA_coll_ibm_collselect_mode_barrier=failsafe ``` was tested and is a tiny bit slower than just falling back to HCOLL or OMPI's barrier implementations via ``` export OMPI_MCA_coll_ibm_skip_barrier=true ``` Thanks to Brian Smith at OLCF for the support! * fix doc compilation on Spock (#2287) Co-authored-by: ax3l <[email protected]> Co-authored-by: ax3l <[email protected]> * CI: EB on Intel DPC++/SYCL (#2279) Compile with embedded boundaries with Intel's `dpcpp` compiler. * Allow Restarts and Checkpoints in python picmi (#2274) * Add a class to output checkpoint diagnostics in picmi * Add argument to picmi Simulation for restarting from a checkpoint * Changes to make the checkpoint work * Change inheritance for Checkpoint and rename variables in Simulation * Remove file exists check * Perlmutter: CMake 3.22 Preview (#2291) Use the new Perlmutter CMake module with the improved `FindMPI.cmake` logic for HPE/Cray that we contributed upstream. This will be part of CMake 3.22 (the Perlmutter module is built from the CMake development branch and contains the patch already). * Fix floating point exception issue in single precision (#2284) * fix fpe issue in single precision * fixed bug * fixed bug * make prettier * fixed bug * fixed bug * fixed bug * using constexpr as suggested by NZaim * Fix uninitialized particle data (#2282) * Include Diagnostics in Evolve Timer (#2278) * Include Diagnostics in Evolve Timer * Update WarpXEvolve.cpp * Update WarpXEvolve.cpp * Update WarpXEvolve.cpp Removed White Spaces * return warpx_py_afterstep and adds comments * Update Source/Evolve/WarpXEvolve.cpp Co-authored-by: Neïl Zaim <[email protected]> * Initialize physical_species (#2273) * warpx_py_beforeEsolve & _afterEsolve with electrostatic solver (#2286) * Various updates related to defining embedded boundaries in Python (#2280) * Various updated related to defining embedded boundaries in Python * Fix typo in comment Co-authored-by: Roelof Groenewald <[email protected]> * Remove unused variable from backtransformed diagnostics (#2296) * Remove unused variable from backtransformed diagnostics * Also remove zmin_lab * Enable particle creation from binary collisions (#2245) * Enable particle creation from binary collisions * Do not directly use DeviceVector * Make defineAllParticleTiles () public * Store and read in (if present) dmaps in checkpoint files (#2248) * Store and read in (if present) dmaps in checkpoint files * also write number of procs to checkpoint * Docs: Spock CMake 3.20.2, ROCm 4.3.0 (#2288) * Docs: Spock CMake 3.20.2 Slightly update CMake to avoid that we run into https://gitlab.kitware.com/cmake/cmake/-/issues/21887 at some point. (Proactively guarding this.) * Spock: ROCm 4.3.0 Update ROCm (tested) * AMReX/PICSAR: Weekly Update (#2299) * PICSAR: Weekly Update * AMReX: Weekly Update * Move QEDEvents() before OneStep functions (#2289) * Move QEDEvents() before OneStep functions * Update BW and QS automated tests * Fix a couple of bugs * Update benchmarks * Forgot to update one benchmark value * Add picmi parameter (#2306) * Set EB (electrostatic) potential with a function of space and time (#2297) * initial commit for adding a spatially varying potential for EBs * removed extra semi-colon * fixed issue causing oneAPI DPC++ build failure * CI: Skip & Fail Draft PRs (#2312) Save resources on PRs marked as "Draft" by skipping & failing CI. * Evolve: Mark Step in Profile (#2308) Mark individual steps in the profiler. This makes it easier to see the individual step timing in our tiny profiler and it also helps to see with NVTX (and equivalent) annotations to find the exact start/end of a time step in traces. * Performance: ApplyStencil w/o Temporaries (#2314) * Performance: ApplyStencil w/o Temporaries Most of the time in this code is spend in allocating/freeing the temporary. Wrap invalid values to zero on the fly instead. * Filter: Fix scomp forwarding Update Source/Filter/Filter.cpp Co-authored-by: Remi Lehe <[email protected]> * Fix: Status CUDA CI Regression from #2312 * Docs: Link Older Developer Tutorial (#2311) Link slides from Maxence in 03/2020. * Rename WarpXWrappers.(h->H) (#2261) * Rename WarpXWrappers.(h->H) All our other header files use `.H`, so renaming this one for consistency. * Remove unused includes of WarpXWrappers.H * AMReX/PICSAR: Weekly Update (#2316) * Fix Bug in PSATD Restart with Time Averaging (#2318) * Fix Bug in PSATD Restart with Time Averaging * PSATD Time Avg Restart: Add CI Test - restart test - use recommended boundary conditions for the spectral solver in z Co-authored-by: Axel Huebl <[email protected]> * Rename eb_potential parameter (#2310) * Python CI: Only 2 Ranks (#2320) Public CI only offers two virtual cores and thus only 2 MPI ranks. Oversubscription generally leads to unstable builds (sporadic suspension of processes by the host system) and newer MPI versions won't start at all without requesting oversubscription. * leveling_thinning: 2 MPI Ranks (#2321) Public CI only offers two virtual cores and thus only 2 MPI ranks. Oversubscription generally leads to unstable builds (sporadic suspension of processes by the host system) and newer MPI versions won't start at all without requesting oversubscription. * Docs: Cori Newer CMake (#2324) Cori deprecated the old CMake module today. ``` Module Deprecation Warning: upgrade to 3.21.3 (new default on 2021-09-21) ``` * External Poisson solver: wrappers to copy the full domain rho and phi between C++ and Python (#2285) * added missing parts to allow an external Poisson solver to be used instead of the MLMG solver * added rho and phi wrappers to fields.py * added an example of using an external Poisson solver * need to run PICMI test on 2 processors same as non-PICMI version in order to compare plotfiles * changes requested during PR review * pass dictionary to eval statement when calculating boundary potential * Move RandomizeVelocity to ParticleUtils (#2326) * Docs: Cori V100 GPU Job Script (#2328) * Docs: Cori V100 GPU Job Script Add a job script template for Cori V100 GPU nodes. * Add concrete cgpu allocation * Update Tools/BatchScripts/batch_cori_gpu.sh Co-authored-by: Axel Huebl <[email protected]> * Update: 1 Rank per GPU - 1 rank per GPU - more doc hints * Slurm: CUDA_VISIBLE_DEVICES Set `--gpus-per-task=1` * Visible devices for real * Doc for GPU Binding Co-authored-by: Remi Lehe <[email protected]> * Initialize the BTD buffers to 0.0 (#2329) * Exchange fewer guard cells with standard PSATD solver (#2266) * Try to exchange fewer guard cells with standard PSATD solver * Add ng_afterPushPSATD variable in GuardCellManager * Print Restart: dt,dx,dy,dz (#2334) Print those quanitites both during setup and after restart. * Fix nvcc compilation bug (#2333) * Stochastic particle reflection from absorbing domain boundaries (#2281) * added tunable particle reflection from absorbing domain boundaries * extended picmi.py to allow setting boundary reflection coefficients and added a CI test for the reflection implementation * allow R(E) to be specified, except for embedded boundaries * changed approach for particle reflection, now the ParticleBoundaries object will hold the reflection coefficient; reflection from EBs not implemented * added functionality to reflect from EB; still needs to be tested for accuracy * added support for energy dependent reflection models for domain boundaries * fixed at least one issue causing CI fails - building reflection model parsers for not physical particle containers * switched reflection coefficients to be functions of the velocity component perpendicular to the boundary rather than energy * reverted initial work on reflecting from EBs * changed naming convention for new CI test for particle reflection * switched useMPI back to 1 in test * breaking changes while trying to sort out GPU issue * fixed issue with CUDA compilation - hopefully :) * various code improvements from PR review suggestions * fix of major issues * no need to parse the reflection models at every step * skip particles that are already flagged for removal in ApplyBoundaryConditions * Python Install Env Vars: All CAPS (#2317) * Python Install Env Vars: All CAPS On Windows, environment variable names are case insensitive and often mapped to all CAPS spelling. - https://stackoverflow.com/questions/7797269/python-environment-case-senstivity-os-environ - https://en.wikipedia.org/wiki/Environment_variable#Syntax This leads to an inconsistency between Unix and Windows like systems if we use case sensitive environment variable names for install control in Python. Luckily, it is quite common to use all-CAPS variables in UNIX for environment controls as well, so we will make a breaking change to unify on that. Existing usage will see a warning of an unused variable with `python3 -m pip install -v .` * setup.py: Caps Python Vars from Env * DPC++: Disable -fPIC (#2338) Sporadic segfaults in `dpcpp` with shared objects since last release. Needs to be triaged. Disabled to unblock community devs. * Restart: Handle Header File Read Errors (#2335) * Restart: Handle Read Errors `std::istringstream` does not throw automatically if errors occur. This leads to silently skipped reads. Enable throwing on all errors to avoid hard to debug follow-up errors. * Fix: Write All Particle Containers Forgot to write laser particle containers but then tried to read them in. The mass & charge of those are not essential for us to restore, but it avoids confusion and extra logic if we just write them properly at the general location. * ParticleIO: Fix All Container Restart (#2341) * ParticleIO: Fix All Container Restart * Fix comment * Doc: Disable CCache (#2339) Document on how to disable CCache. Add CCache control to Python builds. * AMReX: Weekly Update (#2348) ``` ./Tools/Release/updateAMReX.py ``` * Allow more work for OpenMP threads in `WarpX::UpdateAuxilaryData` (#2342) * Allow more work for OpenMP threads in AuxilaryData * Better parallelization in stagger-to-nodal current interpolation * Summit: Known Issues Since RHEL8 (#2349) * Summit: Known Issues Since RHEL8 Document all known issues and work-arounds for Summit users. * Summit: libfabric developer notes * SumBoundary: Use Fewer Guard Cells for J/rho (#2323) * SumBoundary: Use Fewer Guard Cells for J/rho * Account for Stencil of Bilinear Filter * Revert Last Commit, Replace SumBoundary/Copy With ParallelAdd * Account for Stencil of Bilinear Filter * Reset Benchmarks of Averaged Galilean Hybrid Tests * Revert Last Commit This reverts commit 7963176b46708eb08e7c41e67a3b0a6fe6cb2147. * Account for Stencil of Current Centering * Comoving PSATD: New Input psatd.use_default_v_comoving (#2304) * Stencil: Fix non-3D Compiles on GPU (#2353) * CI: HIP 2D Add a HIP 2D Test to compilation. * Stencil: Fix non-3D Compiles on GPU Overlooked the `#else` branch severely. * CI: HIP 2D DP * Update read_raw_data to RZ (with multiple modes) (#2313) * Use correct positions for the moving window of the BTD (#2356) * CI: Update HIP Repo URL (#2359) Thanks to @wyphan in https://github.com/AMReX-Codes/amrex/pull/2366 > This updates the repo line for HIP to the latest version as documented on https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html#installing-a-rocm-package-from-a-debian-repository > Notably, `xenial` is now `ubuntu`. * Update AMReX (#2362) Update pulls in: - Intel DPC++ 2021.4.0 support (SYCL) - Nvidia NVHPC 21.9 support (CUDA) * Rigid injected species: Remove unused function and variables (#2350) * Remove unused function * Remove `focused` and `projected` for rigid particles * Remove array of ints `done_injecting` (#2365) * CMake: More Robust git STRIP (#2358) Sometimes a `git` executable can be found but is unusable. Quoting the result for empty strings makes the build logic more robust in such scenarios. Mitigates: ``` CMake Error at cmake/WarpXFunctions.cmake:289 (string): string sub-command STRIP requires two arguments. Call Stack (most recent call first): CMakeLists.txt:256 (get_source_version) ``` Provoked via: Mixing of brew and Spack (git from brew + Spack env with `export DYLD_LIBRARY_PATH=/Users/axel/src/spack/var/spack/environments/warpx-dev/.spack-env/view/lib`) * Style: Old BTD Definitions (#2352) Add spaces before definitions in the old BTD. Was just adding them as we looked for a solution to fix restarts with the old particle BTD. * Fix restart for rigid injected particles (#2363) * Activate rigid injected species in restart tests * Fix reading of injection position and velocity * Update checksum * Update Examples/Tests/restart/analysis_restart.py * Docs PoP: Fix Co-Authors (#2367) * Implement Checkpoint / Restart for Laser Particles (#2360) * Initialize lasers when restarting + recalculate z position at each iteration * Fix restart for rigid-injected particles * Fix laser injection box at restart * Use correct positions for the moving window of the BTD * implement restart for laser particles * add CI test for laser acceleration restarting * writespace * Revert "Fix restart for rigid-injected particles" This reverts commit 33bf3a60126539b9566ead3115ff2bcdf4b80516. * don't add lasers to all full diags, only checkpoints * fix unused parameter warning * move the call to reset the m_laser_injection_box into PostRestart * bugfix for non-boosted frame simulations * Read the mean velocity of the injected species * Remove unused line * Revert "Initialize lasers when restarting + recalculate z position at each iteration" This reverts commit 322634c35aa996649d0e8fff9fb683d04f2ee121. * Add new ReadHeader / WriteHeader functions * Add WriteHeader/ReadHeader for laser particles * Revert "add CI test for laser acceleration restarting" This reverts commit 6ba0fb3c121027eecf8b5845f55080e5b187375d. * Fix merge commit * Correct reading of updated position * Add laser in restart test * Remove unused parameters * Update regression tests * Add benchmark files * Update boundary conditions for restart_psatd Co-authored-by: Remi Lehe <[email protected]> * Release 21.10 (#2370) * Update: AMReX/PICSAR to 21.10 * WarpX: 21.10 Co-authored-by: Axel Huebl <[email protected]> Co-authored-by: Edoardo Zoni <[email protected]> Co-authored-by: MaxThevenet <[email protected]> Co-authored-by: Neïl Zaim <[email protected]> Co-authored-by: Roelof Groenewald <[email protected]> Co-authored-by: atmyers <[email protected]> Co-authored-by: David Grote <[email protected]> Co-authored-by: Kevin Z. Zhu <[email protected]> Co-authored-by: Phil Miller <[email protected]> Co-authored-by: Luca Fedeli <[email protected]> Co-authored-by: Remi Lehe <[email protected]> Co-authored-by: Corey Wetterer-Nelson <[email protected]> Co-authored-by: Olga Shapoval <[email protected]> Co-authored-by: Luca Fedeli <[email protected]> Co-authored-by: Michael Kieburtz <[email protected]> Co-authored-by: Andrew Myers <[email protected]> Co-authored-by: Peter Scherpelz <[email protected]> Co-authored-by: Edoardo Zoni <[email protected]> Co-authored-by: Yinjian Zhao <[email protected]> Co-authored-by: Tiberius Rheaume <[email protected]> Co-authored-by: Lorenzo Giacomel <[email protected]> Co-authored-by: Weiqun Zhang <[email protected]> Co-authored-by: Phil Miller <[email protected]> Co-authored-by: Roelof <[email protected]> Co-authored-by: Andrei Berceanu <[email protected]> Co-authored-by: ax3l <[email protected]> Co-authored-by: Nicholas Ruof <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
I am using the E4S Spack package of ADIOS2 v2.7.1 on Summit (
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.3.0/adios2-2.7.1-vuplpxvgpbby2oe4sgpbtarlrg55ofxl/). We are really happy about that opportunity, because it saves us from having to maintain and compile ADIOS2 ourselves for the whole WarpX & PIConGPU user base, which eases our support burden a lot and gives us a fast-track to new ADIOS2 releases for production runs.Unfortunately, I think that some components of ADIOS2 forget to declare infiniband (IB) / libfabric or MPI-related dependencies, because the moment I link ADIOS2 to my application (ECP WarpX), I get the following errors on startup of our app, even if I don't use any ADIOS functionality:
I also opened an OLCF ticket for this (OLCFHELP-3319) and linked to here for coordination.
In parallel, it would be perfect if you @pnorbert @chuckatkins @eisenhauer @williamfgc @sklasky et al. can coordinate with OLCF support, your Spack package and E4S to make sure the SST component's CMake logic in combination with the
depends_ondeclarations in your Spack package declare the right dependencies.Furthermore, the Spack ADIOS2 package could declare simple acceptance tests (built-time checks and smoke-tests) to make sure system people can automatically validate your compiled package really works. Those tests can be a small sub-set of your regular
cteststo check MPI et al. truly works.References
ldd libadios2_cxx11_mpi.so
I noticed that
libeventis picked up from the system and not declared asdepends_onin Spack. Maybe that causes this?Also I found
libcrypto.soandlibz.soare not declared dependencies and picked up from the system for/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.3.0/spectrum-mpi-10.4.0.3-20210112-2s7kpbzydf6val7k2d3e6cz3zdhtcwlw/lib/libmpiprofilesupport.so.3ldd warpx
The text was updated successfully, but these errors were encountered: