Update documentation

docs/source/details/backendconfig.rst

@@ -37,7 +37,10 @@ The configuration string may refer to the complete ``openPMD::Series`` or may ad
 This reflects the fact that certain backend-specific parameters may refer to the whole Series (such as storage engines and their parameters) and others refer to actual datasets (such as compression).
 Dataset-specific configurations are (currently) only available during dataset creation, but not when reading datasets.
 
-A JSON/TOML configuration may either be specified as an inline string that can be parsed as a JSON/TOML object, or  alternatively as a path to a JSON/TOML-formatted text file (only in the constructor of ``openPMD::Series``):
+Additionally, some backends may provide different implementations to the ``Series::flush()`` and ``Attributable::flushSeries()`` calls.
+JSON/TOML strings may be passed to these calls as optional parameters.
+
+A JSON/TOML configuration may either be specified as an inline string that can be parsed as a JSON/TOML object, or  alternatively as a path to a JSON/TOML-formatted text file (only in the constructor of ``openPMD::Series``, all other API calls that accept a JSON/TOML specification require in-line datasets):
 
 * File paths are distinguished by prepending them with an at-sign ``@``.
   JSON and TOML are then distinguished by the filename extension ``.json`` or ``.toml``.
@@ -119,9 +122,12 @@ Explanation of the single keys:
   Please refer to the `official ADIOS2 documentation <https://adios2.readthedocs.io/en/latest/engines/engines.html>`_ for the available engine parameters.
   The openPMD-api does not interpret these values and instead simply forwards them to ADIOS2.
 * ``adios2.engine.usesteps``: Described more closely in the documentation for the :ref:`ADIOS2 backend<backends-adios2>` (usesteps).
-* ``adios2.engine.flush_during_step`` Only relevant for BP5 engine, default is ``true``.
-  If true, data will be moved to disk on every flush.
-  If false, then only upon ending an IO step or closing an engine.
+* ``adios2.engine.preferred_flush_target`` Only relevant for BP5 engine, possible values are ``"disk"`` and ``"buffer"`` (default: ``"disk"``).
+
+  * If ``"disk"``, data will be moved to disk on every flush.
+  * If ``"buffer"``, then only upon ending an IO step or closing an engine.
+
+  This behavior can be overridden on a per-flush basis by specifying this JSON/TOML key as an optional parameter to the ``Series::flush()`` or ``Attributable::flushSeries()`` methods.
 * ``adios2.dataset.operators``: This key contains a list of ADIOS2 `operators <https://adios2.readthedocs.io/en/latest/components/components.html#operator>`_, used to enable compression or dataset transformations.
   Each object in the list has two keys:
 

docs/source/usage/workflow.rst

@@ -63,3 +63,12 @@ Attributes are (currently) unaffected by this:
 
 *   In writing, attributes are stored internally by value and can afterwards not be accessed by the user.
 *   In reading, attributes are parsed upon opening the Series / an iteration and are available to read right-away.
+
+.. attention::
+
+    Note that the concrete implementation of ``Series::flush()`` and ``Attributable::seriesFlush()`` is backend-specific.
+    Using these calls does neither guarantee that data is moved to storage/transport nor that it can be accessed by independent readers at this point.
+
+    Some backends (e.g. the BP5 engine of ADIOS2) have multiple implementations for the openPMD-api-level guarantees of flush points.
+    For user-guided selection of such implementations, ``Series::flush`` and ``Attributable::seriesFlush()`` take an optional JSON/TOML string as a parameter.
+    See the section on :ref:`backend-specific configuration <backendconfig>` for details.

include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp

@@ -413,7 +413,7 @@ namespace ADIOS2Defaults
     constexpr const_str str_type = "type";
     constexpr const_str str_params = "parameters";
     constexpr const_str str_usesteps = "usesteps";
-    constexpr const_str str_flushtarget = "flush_target";
+    constexpr const_str str_flushtarget = "preferred_flush_target";
     constexpr const_str str_usesstepsAttribute = "__openPMD_internal/useSteps";
     constexpr const_str str_adios2Schema =
         "__openPMD_internal/openPMD2_adios2_schema";

include/openPMD/Series.hpp

@@ -442,6 +442,11 @@ class Series : public Attributable
     std::string backend() const;
 
     /** Execute all required remaining IO operations to write or read data.
+     *
+     * @param backendConfig Further backend-specific instructions on how to
+     *                      implement this flush call.
+     *                      Must be provided in-line, configuration is not read
+     *                      from files.
      */
     void flush(std::string backendConfig = "{}");
 

include/openPMD/backend/Attributable.hpp

@@ -238,6 +238,11 @@ class Attributable
      * of parents. This method will walk up the parent list until it reaches
      * an object that has no parent, which is the Series object, and flush()-es
      * it.
+     *
+     * @param backendConfig Further backend-specific instructions on how to
+     *                      implement this flush call.
+     *                      Must be provided in-line, configuration is not read
+     *                      from files.
      */
     void seriesFlush(std::string backendConfig = "{}");
 

test/SerialIOTest.cpp

@@ -4025,7 +4025,8 @@ void adios2_bp5_flush(std::string const &cfg, bool flushDuringStep)
             component.resetDataset({Datatype::INT, {size}});
             component.storeChunk(data, {0}, {size});
             // component.seriesFlush(FlushMode::NonCollective);
-            component.seriesFlush("adios2.engine.flush_target = \"buffer\"");
+            component.seriesFlush(
+                "adios2.engine.preferred_flush_target = \"buffer\"");
         }
         // everything should be the same as before
         currentSize = getsize();
@@ -4047,7 +4048,8 @@ void adios2_bp5_flush(std::string const &cfg, bool flushDuringStep)
             component.resetDataset({Datatype::INT, {size}});
             component.storeChunk(data, {0}, {size});
             // component.seriesFlush(FlushMode::NonCollective);
-            component.seriesFlush("adios2.engine.flush_target = \"disk\"");
+            component.seriesFlush(
+                "adios2.engine.preferred_flush_target = \"disk\"");
         }
         currentSize = getsize();
         // should now indiscriminately be roughly within 1% of 16Gb
@@ -4067,7 +4069,7 @@ TEST_CASE("adios2_bp5_flush", "[serial][adios2]")
 [adios2.engine]
 usesteps = true
 type = "bp5"
-flush_target = "disk"
+preferred_flush_target = "disk"
 
 [adios2.engine.parameters]
 AggregationType = "TwoLevelShm"
@@ -4085,7 +4087,7 @@ BufferChunkSize = 2147483646 # 2^31 - 2
 [adios2.engine]
 usesteps = true
 type = "bp5"
-flush_target = "buffer"
+preferred_flush_target = "buffer"
 
 [adios2.engine.parameters]
 AggregationType = "TwoLevelShm"
@@ -4103,7 +4105,7 @@ BufferChunkSize = 2147483646 # 2^31 - 2
 [adios2.engine]
 usesteps = true
 type = "bp5"
-# flush_target = <default>
+# preferred_flush_target = <default>
 
 [adios2.engine.parameters]
 AggregationType = "TwoLevelShm"