Update documentations for the latest ngen

INSTALL.md

@@ -60,18 +60,20 @@ In addition to normal CMake options, the following `ngen` configuration options
 
 Option                | Description
 --------------------- | -----------
-NETCDF_ACTIVE         | Include NetCDF support
+NGEN_WITH_NETCDF      | Include NetCDF support
 NGEN_WITH_SQLITE3     | Include SQLite3 support (GeoPackage support)
-UDUNITS_ACTIVE        | Include UDUNITS support
-MPI_ACTIVE            | Include MPI (Parallel Execuation) support
-BMI_FORTRAN_ACTIVE    | Include Fortran BMI support
-BMI_C_LIB_ACTIVE      | Include C BMI support
-NGEN_ACTIVATE_PYTHON  | Include Python support
-NGEN_ACTIVATE_ROUTING | Include `t-route` integration
+NGEN_WITH_UDUNITS     | Include UDUNITS support
+NGEN_WITH_MPI         | Include MPI (Parallel Execuation) support
+NGEN_WITH_BMI_FORTRAN | Include Fortran BMI support
+NGEN_WITH_BMI_C       | Include C BMI support
+NGEN_WITH_PYTHON      | Include Python support
+NGEN_WITH_ROUTING     | Include `t-route` integration
+NGEN_WITH_TESTS       | Include unit and integration tests
+NGEN_QUIET            | Include suppressing unwanted message output
 
-> These can be included by adding `-D{Option}=[ON/OFF]` to the following CMake command.
+> These can be included by adding `-D{Option:BOOL}=[ON/OFF]` to the following CMake command.
 
-> **Note**: if `-DNGEN_ACTIVATE_PYTHON=ON`, and you are using an *activated* virtual environment, then CMake will bind its Python configuration to your virtual environment. This means, when you run the `ngen` executable, your virtual environment must be activated.
+> **Note**: if `-DNGEN_WITH_PYTHON:BOOL=ON`, and you are using an *activated* virtual environment, then CMake will bind its Python configuration to your virtual environment. This means, when you run the `ngen` executable, your virtual environment must be activated.
 
 The following CMake command will configure the build:
 
@@ -105,4 +107,4 @@ cmake --build build --target ngen -- -j 2
 
 **Further information on each step:**
 
-Make sure all necessary [dependencies](doc/DEPENDENCIES.md) are installed, and then [build the main ngen target with CMake](doc/BUILDS_AND_CMAKE.md).  Then run the executable, as described [here for basic use](README.md#usage) or [here for distributed execution](doc/DISTRIBUTED_PROCESSING.md#examples).
+Make sure all necessary [dependencies](doc/DEPENDENCIES.md) are installed, and then [build the main ngen target with CMake](doc/BUILDS_AND_CMAKE.md).  Then run the executable, as described [here for basic use](README.md#usage) or [here for distributed execution](doc/DISTRIBUTED_PROCESSING.md#examples). A step by step [instruction](https://github.com/NOAA-OWP/ngen/wiki/NGen-Tutorial) and install with more [options](https://github.com/NOAA-OWP/ngen/wiki/Building) are also provided on our wiki page.

README.md

@@ -77,11 +77,11 @@ For example:
 Or, if the build system has not yet been properly generated:
 
     git submodule update --init --recursive -- test/googletest
-    cmake -DCMAKE_BUILD_TYPE=Debug -B cmake-build-debug -S .
+    cmake -DCMAKE_BUILD_TYPE=Debug -DNGEN_WITH_TESTS:BOOL=ON -B cmake-build-debug -S .
     cmake --build cmake-build-debug --target test_all -- -j 4
     ./cmake-build-debug/test/test_all
 
-See the [Testing ReadMe](test/README.md) file for a more thorough discussion of testing.
+See the [Testing ReadMe](test/README.md) file and [wiki/Quickstart](https://github.com/NOAA-OWP/ngen/wiki/NGen-Tutorial) for a more thorough discussion of testing.
 
 ## How to debug the software
 

data/gauge_01073000/routing_config.yaml

@@ -8,21 +8,36 @@ network_topology_parameters:
     #----------
     supernetwork_parameters:
         #----------
-        geo_file_type: HYFeaturesNetwork 
+        network_type: HYFeaturesNetwork 
         geo_file_path: data/gauge_01073000/gauge_01073000.gpkg 
         mask_file_path: # domain/unit_test_noRS/coastal_subset.txt
         synthetic_wb_segments:
             #- 4800002
             #- 4800004
             #- 4800006
             #- 4800007
+        columns: 
+            key: 'id'
+            downstream: 'toid'
+            dx : 'length_m'
+            n : 'n'
+            ncc : 'nCC'
+            s0 : 'So'
+            bw : 'BtmWdth'
+            waterbody : 'rl_NHDWaterbodyComID'
+            gages : 'rl_gages'
+            tw : 'TopWdth'
+            twcc : 'TopWdthCC'
+            musk : 'MusK'
+            musx : 'MusX'
+            cs : 'ChSlp'
+            alt: 'alt'
     waterbody_parameters:
         #----------
         break_network_at_waterbodies: False 
         level_pool:
             #----------
-            level_pool_waterbody_parameter_file_path: data/gauge_01073000/gauge_01073000.gpkg 
-            reservoir_parameter_file                : data/gauge_01073000/gauge_01073000.gpkg
+            level_pool_waterbody_parameter_file_path: data/gauge_01073000/gauge_01073000.gpkg
         #rfc:
             #----------
             #reservoir_parameter_file                : domain/reservoir_index_AnA.nc
@@ -60,9 +75,7 @@ compute_parameters:
         dt                          : 300 # [sec]
         qlat_input_folder           : ./ 
         qlat_file_pattern_filter    : "nex-*"
-        nexus_input_folder          : ./ 
-        nexus_file_pattern_filter   : "nex-*" #OR "*NEXOUT.parquet" OR "nex-*"
-        binary_nexus_file_folder    : ./ # this is required if nexus_file_pattern_filter="nex-*"
+        binary_nexus_file_folder    : ./ # this is required if qlat_file_pattern_filter="nex-*"
         #coastal_boundary_input_file : channel_forcing/schout_1.nc  
         nts                         : 8640 #288 for 1day
         max_loop_size               : 720 # [hr]  
@@ -81,11 +94,15 @@ compute_parameters:
     #        crosswalk_segID_field         : # 'link'
     #        wrf_hydro_lastobs_file        :                
     #        lastobs_output_folder         : lastobs/
-    #    reservoir_da:
-    #        #----------
-    #        reservoir_persistence_usgs  : False
-    #        reservoir_persistence_usace : False
-    #        gage_lakeID_crosswalk_file  : # domain/reservoir_index_AnA.nc
+        reservoir_da:
+            #----------
+            reservoir_persistence_da:
+                #----------
+                reservoir_persistence_usgs  : False
+                reservoir_persistence_usace : False
+            reservoir_rfc_da:
+                #----------
+                reservoir_rfc_forecasts: False
 #--------------------------------------------------------------------------------
 output_parameters:
     #----------

doc/BMI_MODELS.md

@@ -188,7 +188,7 @@ There are some special BMI formulation config parameters which are required in c
 
 For **C** models, the model must be packaged as a pre-compiled shared library.  A CMake cache variables can be configured for controlling whether the framework functionality for working with BMI C libraries is activated.  This is found in, or must be added to, the _CMakeCache.txt_ file in the build system directory:
 
-* `BMI_C_LIB_ACTIVE`
+* `NGEN_WITH_BMI_C`
   * type: `BOOL`
   * must be set to `ON` (or equivalent in CMake) for BMI C shared library functionality to be compiled and active
 
@@ -213,9 +213,9 @@ An example implementation for an appropriate BMI model as a **C** shared library
 
 #### BMI C Activate/Deactivation Required in CMake Build
 
-BMI C functionality will not work (i.e., will not be compiled or executable) unless set to be active in the CMake build.  This requires setting the `BMI_C_LIB_ACTIVE` CMake cache variable to `ON` or `TRUE` (or equivalent).
+BMI C functionality will not work (i.e., will not be compiled or executable) unless set to be active in the CMake build.  This requires setting the `NGEN_WITH_BMI_C` CMake cache variable to `ON`.
 
-Conversely, built executables (and perhaps certain build targets) may not function as expected if `BMI_C_LIB_ACTIVE` is `ON` but the configured shared library is not available.
+Conversely, built executables (and perhaps certain build targets) may not function as expected if `NGEN_WITH_BMI_C` is `ON` but the configured shared library is not available.
 
 #### Additional Bootstrapping Function Needed
 
@@ -359,7 +359,7 @@ An example implementation for an appropriate BMI model as a **Python** class is
 
 ### Enabling Fortran Integration
 
-To enable Fortran integration functionality, the CMake build system has to be [generated](BUILDS_AND_CMAKE.md#generating-a-build-system) with the `NGEN_BMI_FORTRAN_ACTIVE` CMake variable set to `ON`.
+To enable Fortran integration functionality, the CMake build system has to be [generated](BUILDS_AND_CMAKE.md#generating-a-build-system) with the `NGEN_WITH_BMI_FORTRAN` CMake variable set to `ON`.
 
 ### ISO C Binding Middleware
 Nextgen takes advantage of the Fortran `iso_c_binding` module to achieve interoperability with Fortran modules.  In short, this works through use of an intermediate middleware module maintained within Nextgen.  This module handles the ([majority of the](#required-additional-fortran-registration-function)) binding through proxy functions that make use of the actual external BMI Fortran module.  

doc/DEPENDENCIES.md

@@ -6,15 +6,16 @@
 | ------------- |:------------- | :-----:| :-------: |
 | [Google Test](#google-test) | submodule  | `release-1.10.0` | |
 | [C/C++ Compiler](#c-and-c-compiler) | external | see below |  |
-| [CMake](#cmake) | external | \>= `3.14` | |
+| [CMake](#cmake) | external | \>= `3.17` | |
 | [Boost (Headers Only)](#boost-headers-only) | external | `1.79.0` | headers only library |
 | [Udunits libraries](https://www.unidata.ucar.edu/software/udunits) | external | >= 2.0 | Can be installed via package manager or from source |
 | [MPI](https://www.mpi-forum.org) | external | No current implementation or version requirements | Required for [multi-process distributed execution](DISTRIBUTED_PROCESSING.md) |
-| [Python 3 Libraries](#python-3-libraries) | external | \> `3.6.8` | Can be [excluded](#overriding-python-dependency). |
+| [Python 3 Libraries](#python-3-libraries) | external | \>= `3.8.0` | Can be [excluded](#overriding-python-dependency). |
 | [pybind11](#pybind11) | submodule | `v2.6.0` | Can be [excluded](#overriding-pybind11-dependency). |
 | [dmod.subsetservice](#the-dmodsubsetservice-package) | external | `>= 0.3.0` | Only required to perform integrated [hydrofabric file subdividing](DISTRIBUTED_PROCESSING.md#subdivided-hydrofabric) for distributed processing . |
 | [t-route](#t-route) | submodule | see below | Module required to enable channel-routing.  Requires pybind11 to enable |
-
+| [NetCDF Libraries](#netcdf-libraries) | external | \>= `4.7.4` | Enables NetCDF I/O support |
+| [SQLite3](https://www.sqlite.org/cintro.html) | external | \> `3.7.17` | Enables GeoPackage reading support |
 # Details
 
 ## Google Test
@@ -219,3 +220,10 @@ Be sure to build ngen with Python and Routing support as discussed there, and if
 ```sh
 source venv/bin/activate
 ```
+
+## NetCDF Libraries
+
+### Setup
+
+First, check if your compute system already have a version that is up to date, usually in the sub-directories (include/ and lib64/) under /usr. If not, you will have to install your own version. For instructions on how to install netCDF, some randomly selected references are [here](https://docs.unidata.ucar.edu/nug/current/getting_and_building_netcdf.html) and [here](https://docs.geoserver.org/main/en/user/extensions/netcdf-out/nc4.html).
+

doc/DISTRIBUTED_PROCESSING.md

@@ -26,7 +26,7 @@ When executed, the driver is provided a valid partitioning configuration file pa
 
 # Building the MPI-Enabled Nextgen Driver
 
-To enable distributed processing capabilities in the driver, the [CMake build must be generated](BUILDS_AND_CMAKE.md) with the `MPI_ACTIVE` variable set to `TRUE` or `ON`, and the necessary MPI libraries must be available to CMake.  Additionally, [certain features](#on-the-fly-generation) are only supported if `NGEN_ACTIVATE_PYTHON` is also `ON`.
+To enable distributed processing capabilities in the driver, the [CMake build must be generated](BUILDS_AND_CMAKE.md) with the `NGEN_WITH_MPI` variable set to `ON`, such as `NGEN_WITH_MPI:BOOL=ON` and the necessary MPI libraries must be available to CMake.  Additionally, [certain features](#on-the-fly-generation) are only supported if `NGEN_WITH_PYTHON` is also `ON`.
 
 Otherwise, if using the standard build process with CMake, the driver is built using the same commands.  It can be built either as part of the entire project or the `ngen` CMake target.  CMake manages the necessary adjustments to, e.g., the compiler used, etc. 
 
@@ -56,7 +56,7 @@ The name of an expected or generated subdivided hydrofabric file is based on two
 This will have a partition specific suffix but otherwise have the same name as the full hydrofabric files.  E.g., _catchment_data.geojson.0_ would be the subdivided hydrofabric file for _catchment_data.geojson_ specific to rank 0.  
 
 ### On-the-fly Generation
-Driver processes may, under certain conditions, be able to self-subdivide a hydrofabric and generate the files when necessary.  For this to be possible, the executable must have been built with Python support (via the CMake `NGEN_ACTIVATE_PYTHON` being set to `ON`), and the [required package](DEPENDENCIES.md#the-dmodsubsetservice-package) must be installed within the Python environment available to the driver processes.
+Driver processes may, under certain conditions, be able to self-subdivide a hydrofabric and generate the files when necessary.  For this to be possible, the executable must have been built with Python support (via the CMake `NGEN_WITH_PYTHON` being set to `ON`), and the [required package](DEPENDENCIES.md#the-dmodsubsetservice-package) must be installed within the Python environment available to the driver processes.
 
 ## Examples
 
@@ -66,7 +66,7 @@ Driver processes may, under certain conditions, be able to self-subdivide a hydr
 * the catchment and nexus hydrofabric files, realization config, and partition config have intuitive names and are located in the current working directory
 * all processes completely load the entire hydrofabric
 ```
-mpirun -n 4 cmake-build/ngen catchment_data.geojson "" nexus_data.geojson "" realization_config.json partition_config.json
+mpirun -n 4 cmake-build/ngen catchment_data.geojson "all" nexus_data.geojson "all" realization_config.json partition_config.json
 ```
 
 ### Example 2 - Subdivided Hydrofabric
@@ -75,7 +75,7 @@ mpirun -n 4 cmake-build/ngen catchment_data.geojson "" nexus_data.geojson "" rea
 * the catchment and nexus hydrofabric files, realization config, and partition config have intuitive names and are located in the current working directory
 * each process only load files for a subdivided portion of the hydrofabric that corresponds to a given process's partition
 ```
-mpirun -n 8 cmake-build/ngen catchment_data.geojson "" nexus_data.geojson "" realization_config.json partition_config.json --subdivided-hydrofabric
+mpirun -n 8 cmake-build/ngen catchment_data.geojson "all" nexus_data.geojson "all" realization_config.json partition_config.json --subdivided-hydrofabric
 ```
 
 # Partitioning Config Generator

doc/MPI_REMOTE_NEXUS.md

@@ -10,7 +10,7 @@ The basic outline of steps needed to run the Remote Nexus Class with MPI is:
   * Install MPI.
   * Create the build directory including the option to activate MPI: 
 
-      `cmake -DCMAKE_BUILD_TYPE=Debug -B cmake-build-debug -DMPI_ACTIVE:BOOL=ON -S .`  
+      `cmake -DCMAKE_BUILD_TYPE=Debug -B cmake-build-debug -DNGEN_WITH_MPI:BOOL=ON -DNGEN_WITH_TESTS:BOOL=ON -S .`  
 
   * Unit tests for the Remote Nexus Class can then be built and run from the main directory with the following two commands:
 

doc/PYTHON_ROUTING.md

@@ -49,19 +49,26 @@ FC=mpif90 ./compiler.sh
 
 The `compiler.sh` script will install the Python modules with `-e`. On macOS, you may need to re-install the modules in t-route's `src` directory directly after running `compiler.sh`.
 
+### Some tips on installation if you run into issues
+  * [On install mpi4py](https://github.com/Unidata/netcdf4-python/issues/1296)
+
+  * [On pip version](https://github.com/NOAA-OWP/t-route/issues/621)
+
+  * [On NetCDF version](https://github.com/NOAA-OWP/t-route/issues/705)
+
 [Additional documentation for configuration and dependencies of t-route](https://github.com/NOAA-OWP/t-route#configuration-and-dependencies).
 
 ## Using t-route with ngen
   * Create the build directory including the options to activate Python and Routing: 
 
-      * Activate Python flag with `-DNGEN_ACTIVATE_PYTHON:BOOL=ON`
+      * Activate Python flag with `-DNGEN_WITH_PYTHON:BOOL=ON`
 
-      * Activate Routing flag with `-DNGEN_ACTIVATE_ROUTING:BOOL=ON.`  
+      * Activate Routing flag with `-DNGEN_WITH_ROUTING:BOOL=ON.`  
 
       * An example create build directory command with the above two options activated:
 
     ```sh
-      cmake -B cmake_build -DNGEN_ACTIVATE_PYTHON:BOOL=ON -DNGEN_ACTIVATE_ROUTING:BOOL=ON .
+      cmake -B cmake_build -DNGEN_WITH_PYTHON:BOOL=ON -DNGEN_WITH_ROUTING:BOOL=ON -DNGEN_WITH_TESTS:BOOL=ON .
     ```  
 
   * Unit tests for the Routing_Py_Adapter class can then be built and run from the main directory with the following two commands:

doc/REALIZATION_CONFIGURATION.md

@@ -6,7 +6,7 @@ A Realization Configuration needs to be in [JSON format (JavaScript Object Notat
 
 The Configuration is a key-value object and must contain these three first level object keys:
 * `global` 
-  * is a key-value object that must include an object key for `formulations` that defines the default formulation(s) and also an object key for `forcing` that defines the default forcing file name pattern and path for any catchment that is not defined in `catchments`
+  * is a key-value object that must include an object key for `formulations` that defines the default formulation(s) and also an object key for `forcing` that defines the default forcing file name pattern, path, and provider for any catchment that is not defined in `catchments`
   * Note: `global` can be omitted only if every catchment is assigned a formulation 
 
 * `time`
@@ -32,7 +32,7 @@ The `global` key-value object must contain the following two object keys:
   * Note: future versions could support breaking up `params` into additional key-value subobjects for `options` and `initial_conditions`
   * `params` must be a list that holds key-value pairs
 * `forcing`
-  * key-value object with keys for `file_pattern` and `path` that define the default CSV file pattern and path for the input forcings relative to the executable directory
+  * key-value object with keys for `file_pattern` and `path` that define the default CSV file pattern and path for the input forcings relative to the executable directory. More recently, `ngen` developed the capability to handle forcing data in different formats. Thus, a `provider` value parameter can be used to explicitly define the format of the forcing data, such as NetCDF format, in the form "provider": "NetCDF".
 
 ```
 "global": {

test/README.md

@@ -28,9 +28,9 @@ The CMake buildsystem will need to either be generated or regenerated after the
 
 Regenerating just requires an initial step of removing the/an existing CMake build directory (frequently something like `cmake-build-debug/` in the project root). 
 
-If/once the desired build directory does not exist, use an appropriate CMake command to generate the buildsystem.  E.g.:
+If/once the desired build directory does not exist, use an appropriate CMake command to generate the buildsystem (see [wiki/Quickstart](https://github.com/NOAA-OWP/ngen/wiki/NGen-Tutorial)) for more detailed description.  E.g.:
 
-    cmake -DCMAKE_BUILD_TYPE=Debug -B cmake-build-debug -S .
+    cmake -DCMAKE_BUILD_TYPE=Debug -DNGEN_WITH_TESTS:BOOL=ON -B cmake-build-debug -S .
 
 # Executing Automated Tests