Merge pull request #199 from WireCell/check-test

Improve test situation

.gitignore

@@ -66,9 +66,13 @@ ltximg
 auto
 latexmk-out
 *.tex
-
 svg-inkscape
 *.html
+tmp
+*.tar
+*.md
+README.tex
 larsoft-cfg
 *.dot
 configure
+

README.org

@@ -1,38 +1,67 @@
 #+TITLE: Wire-Cell Toolkit 
+#+SETUPFILE: setup-readme.org
 
-Welcome to the Wire-Cell Toolkit source repository.
+Welcome to the Wire-Cell Toolkit (WCT) source repository at https://github.com/wirecell/wire-cell-toolkit.
 
-See http://wirecell.bnl.gov/ for documentation including user manual and news "blog".
+* Overview
 
-[[https://travis-ci.org/WireCell/wire-cell-toolkit][https://api.travis-ci.org/WireCell/wire-cell-toolkit.svg?branch=master]]
+The WCT is a multi-faceted, high performance software project developed for liquid argon time projection chamber (LArTPC) simulation and data processing.  Some features of wCT include:
+
+- Layered tool design culminating in a modular execution policy and reference command-line interface program.
+- A multi-threaded execution model constructed following the data flow programming paradigm.
+- Plugin, factory and configuration subsystems.
+- Components providing simulation, signal processing and physics reconstruction algorithms.
+- Suite of abstract interface classes.
+- Low level utility algorithms, data structures and streaming data I/O formats.
+
+Additional "README" information is available in the WCT sub packages:
+
+- [[file:apps/README.org][apps]]
+- [[file:aux/README.org][aux]]
+- [[file:cfg/README.org][cfg]]
+- [[file:cuda/README.org][cuda]]
+- [[file:gen/README.org][gen]]
+- [[file:iface/README.org][iface]]
+- [[file:img/README.org][img]]
+- [[file:pgraph/README.org][pgraph]]
+- [[file:pytorch/README.org][pytorch]]
+- [[file:root/README.org][root]]
+- [[file:sigproc/README.org][sigproc]]
+- [[file:sio/README.org][sio]]
+- [[file:tbb/README.org][tbb]]
+- [[file:test/README.org][test]]
+- [[file:util/README.org][util]]
+- [[file:waft/README.org][waft]]
+- [[file:zio/README.org][zio]]
+
+See http://wirecell.bnl.gov/ for the home of Wire-Cell Toolkit documentation and news "blog".
 
 
 * Installation
 
-Wire-Cell Toolkit provides simple and automatic installation which
-gives you, the installer, some options.  
+Wire-Cell Toolkit provides simple and automated installation while allowing you to adapt it so you may provide the required dependencies in a variety of ways.
 
 ** External software dependencies
 
-The WCT dependency tree:
+The WCT dependencies are curated and minimized with some required and some optional.  Below shows the intra- and inter-package dependency tree:
 
 [[file:wct-deps.png]]
 
-Anything that the ~WireCellUtil~ package depends on is required.  The
-rest are optional.  Missing optional dependencies will cause the
-dependent WCT package to not be built.
+Black arrows are library dependencies, blue are for applications and gray are for testing programs.    They represent compile/link time dependencies.
+
+The dependencies for the ~WireCellUtil~ package are required.  The rest are optional.  Missing optional dependencies, or ones specifically turned off, will cause the dependent WCT package to not be built.
 
 Some external dependencies have explicit minimum required versions:
 
 - TBB (oneAPI) 2021.1.1
 - Boost 1.75.0
 
-You may provide the necessary external software dependencies in a
-manner of your own choosing and some options include:
+You may provide the necessary external software dependencies in a manner of your own choosing and some options include:
 
+- Packages provided by your OS or built "by hand".
 - [[https://github.com/WireCell/wire-cell-spack][Spack-based install]] automatically builds all (non-OS) externals and WCT itself
 - Some WCT releases are built at FNAL as a UPS product named =wirecell=.
-- Exploit the above with a [[https://github.com/WireCell/wire-cell-singularity][Singularity container and CVMFS]] (currently recommended)
+- Exploit the above with a [[https://github.com/WireCell/wire-cell-singularity][Singularity container and CVMFS]].
 
 ** Developer Source
 
@@ -44,9 +73,7 @@ Developers check out =master= branch via SSH.
 
 ** User Source
 
-Users typically should build a release branch, either the tip or a
-tagged release on that branch.  Tagged releases are shown on the [[https://github.com/WireCell/wire-cell-toolkit/releases][this
-GitHub release page]].  
+Users typically should build a release branch, either the tip or a tagged release on that branch.  Tagged releases are shown on the [[https://github.com/WireCell/wire-cell-toolkit/releases][this GitHub release page]].
 
 Users may also anonymously clone in the usual way:
 
@@ -62,16 +89,13 @@ On well-behaved systems, configuring the source may be as simple as:
   $ ./wcb configure --prefix=/path/to/install
 #+END_EXAMPLE
 
-Software dependencies which can not automatically be located in system
-areas or via ~pkg-config~ can be manually specified.  For a list of
-options run:
+Software dependencies which can not automatically be located in system areas or via ~pkg-config~ can be manually specified.  For a list of options run:
 
 #+BEGIN_EXAMPLE
   $ ./wcb --help
 #+END_EXAMPLE
 
-Here is an example where some packages are found automatically and
-some need help and others are explicitly turned off:
+Here is an example where some packages are found automatically and some need help and others are explicitly turned off:
 
 #+begin_example
   $ ./wcb configure \
@@ -89,69 +113,42 @@ some need help and others are explicitly turned off:
 
 ** Building
 
-It is suggested to first build the code before running tests.
+The libraries and programs may be built with:
 
 #+BEGIN_EXAMPLE
-  $ ./wcb -p --notests
+  $ ./wcb
 #+END_EXAMPLE
 
 ** Installing
 
-To install the built toolkit and its configuration support files while
-still avoiding the tests do:
+To install:
 
 #+BEGIN_EXAMPLE
-  $ ./wcb -p --notests install
+  $ ./wcb install
 #+END_EXAMPLE
 
-Optionally, the *reference* configuration and data files for one or more
-supported experiments may be installed by giving naming them with the
-~--install-config~ option.  A name matches a sub-directory under
-[[file:cfg/pgrapher/experiment/][cfg/pgrapher/experiment/]] or the special ~all~ name will install all.
+Optionally, the *reference* configuration and data files for one or more supported experiments may be installed by giving naming them with the ~--install-config~ option.  A name matches a sub-directory under [[file:cfg/pgrapher/experiment/][cfg/pgrapher/experiment/]] or the special ~all~ name will install all.
 
 #+begin_example
-  $ ./wcb -p --notests --install-config=<name> install
+  $ ./wcb --install-config=<name> install
 #+end_example
 
 ** Testing
 
-Running the tests can take a while but should be run on new
-installations and after any significant development.  The developers
-try not to leave broken tests so any test failure should be treated as
-important.  However, some tests require proper environment to run
-successfully.  In particular, tests need to find Wire-Cell
-configuration and the shared libraries of the external software
-dependencies need to be located.  Below shows an example:
+Running the tests can take a while but are off by default.  They may be run with:
+#+begin_example
+  $ ./wcb --tests
+#+end_example
 
-#+BEGIN_EXAMPLE
-  $ export WIRECELL_PATH=$HOME/dev/wct/wire-cell-data:$HOME/dev/wct/wire-cell-toolkit/cfg
-  $ export LD_LIBRARY_PATH=$HOME/dev/wct/install/lib:$HOME/opt/jsonnet/lib
-  $ ./wcb -p --alltests
-  ...
-  execution summary 
-    tests that pass 83/83
-      ... 
-    tests that fail 0/83 
-  'build' finished successfully (15.192s)
-#+END_EXAMPLE
+See [[file:tests/README.org]] for more details on testing.
 
 * Release management
 
-To make releases, the above details are baked into two test scripts
-[[https://github.com/WireCell/waf-tools/blob/master/make-release.sh][make-release.sh]] and [[https://github.com/WireCell/waf-tools/blob/master/test-release.sh][test-release.sh]].  See comments at the top of each
-for how to run them.  These scripts can be used by others but are
-meant for developers to make official releases.
-
-* Meta
-
-A new =wcb= build script is made from [[https://github.com/waf-project/waf][waf source]] via:
+WCT uses an ~X.Y.Z~ version string.  While ~X=0~, a ~0.Y.0~ version indicates a new release that may extend or break API or ABI compared to ~Y-1~.  A ~Z>0~ indicates a bug fix to ~Z-1~ which should otherwise retain the API and ABI.  Bug fixes will be made on a branch rooted on ~0.X.0~ called ~0.X.x~.
 
-#+BEGIN_EXAMPLE
-  $ cd waf-tools
-  $ ./refresh-wcb -o /path/to/wire-cell-toolkit/wcb
-  $ cd /path/to/wire-cell-toolkit/
-  $ git commit -am "update wcb" && git push
-#+END_EXAMPLE
+To make releases, the above details are baked into two test scripts [[https://github.com/WireCell/waf-tools/blob/master/make-release.sh][make-release.sh]] and [[https://github.com/WireCell/waf-tools/blob/master/test-release.sh][test-release.sh]].  See comments at the top of each for how to run them.  These scripts can be used by others but are meant for developers to make official releases.
 
+* Meta
 
+Prior to 0.25.0, ~wcb~ was a custom version of [[https://waf.io][Waf]] and is now simply a copy of ~waf~.  The customized tools are held in the [[file:waft/]] directory.
 

apps/README.org

@@ -0,0 +1,136 @@
+#+title: Wire Cell Toolkit Apps
+#+SETUPFILE: ../setup-readme.org
+
+
+* Overview
+
+This package provides toolkit layers at the top of an application
+stack.  This includes
+
+- main command line programs, in particular ~wire-cell~.
+
+- a ~Main~ class used in CLI and other application interfaces (such as /art/).
+
+- a few so called "apps" which are WCT components that provide a
+  high-level execution policy.  See also ~TbbFlow~ in sub package ~tbb~
+  and ~Pgrapher~ in
+
+* Programs
+
+** ~wire-cell~
+
+The ~wire-cell~ command line program provides a "reference" application
+of the toolkit.  It is a generic, "policy free" program that is fully
+driven by configuration.
+
+#+begin_src bash :results output
+  wire-cell --help
+  wire-cell --version
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Command line interface to the Wire-Cell Toolkit
+
+Usage:
+	wire-cell [options] [configuration ...]
+
+Options:
+  -h [ --help ]         produce help message
+  -l [ --logsink ] arg  set log sink as <filename> or 'stdout' or 'stderr', a 
+                        log level for the sink may be given by appending 
+                        ':<level>'
+  -L [ --loglevel ] arg set lowest log level for a log in form 'name:level' or 
+                        just give 'level' value for all (level one of: 
+                        critical,error,warn,info,debug,trace)
+  -a [ --app ] arg      application component to invoke
+  -c [ --config ] arg   provide a configuration file
+  -p [ --plugin ] arg   specify a plugin as name[:lib]
+  -V [ --ext-str ] arg  specify a Jsonnet external variable=<string>
+  -C [ --ext-code ] arg specify a Jsonnet external variable=<code>
+  -A [ --tla-str ] arg  specify a Jsonnet top level arguments variable=<string>
+  --tla-code arg        specify a Jsonnet top level arguments variable=<code>
+  -P [ --path ] arg     add to JSON/Jsonnet search path
+  -t [ --threads ] arg  limit number of threads used
+  -v [ --version ]      print the compiled version to stdout
+
+0.24.0-33-gf9d92c77
+#+end_example
+
+** ~wcsonnet~
+
+The ~wcsonnet~ program is a thin wrapper around the Jsonnet library used to build WCT.  It can be preferable to the standard ~jsonnet~ program for the following reasons:
+
+- It uses the Go Jsonnet library which is substantially faster than the C/C++ library used by ~jsonnet~.
+- It honors the ~WIRECELL_PATH~ to locate files.
+
+#+begin_src shell :results output
+  wcsonnet --help
+#+end_src
+
+#+RESULTS:
+#+begin_example
+wcsonnet is a Wire-Cell Toolkit aware Jsonnet compiler
+Usage: wcsonnet [OPTIONS] [file]
+
+Positionals:
+  file TEXT                   Jsonnet file to compile
+
+Options:
+  -h,--help                   Print this help message and exit
+  -o,--output TEXT            Output file
+  -P,--path TEXT ...          Search paths to consider in addition to those in WIRECELL_PATH
+  -V,--ext-str TEXT ...       Jsonnet external variable as <name>=<string>
+  -C,--ext-code TEXT ...      Jsonnet external code as <name>=<string>
+  -A,--tla-str TEXT ...       Jsonnet level argument value <name>=<string>
+  -S,--tla-code TEXT ...      Jsonnet level argument code <name>=<string>
+
+#+end_example
+
+** ~wcwires~
+
+One of the main input configurations to many WCT algorithms is the
+"wire geometry".  This is typically an exhaustive list of wire (or
+strip) endpoints and their channel and other identifiers.  In many
+cases, the "wires files" are provided with errors.  They may not
+follow correct ordering conventions or they may have poor precision in
+wire endpoints.  WCT provides a way to validate and correct the wire
+geometry when a "wires file" is read in and ~wcwires~ provides this
+functionality in a convenient command line interface.
+
+#+begin_src shell :results output
+  wcwires --help
+#+end_src
+
+#+RESULTS:
+#+begin_example
+wcwires converts and validates Wire-Cell Toolkit wire descriptions
+Usage: wcwires [OPTIONS] [file]
+
+Positionals:
+  file TEXT                   wires file
+
+Options:
+  -h,--help                   Print this help message and exit
+  -P,--path TEXT ...          Search paths to consider in addition to those in WIRECELL_PATH
+  -o,--output TEXT            Write out a wires file (def=none)
+  -c,--correction INT         Correction level: 1=load,2=order,3=direction,4=pitch (def=4)
+  -v,--validate               Perform input validation (def=false)
+  -f,--fail-fast              Fail on first validation error (def=false)
+  -e,--epsilon FLOAT          Unitless relative error determining imprecision during validation (def=1e-6)
+
+#+end_example
+
+
+* WCT ~Main~
+
+WCT provides a C++ class called ~Main~ which may be used to easily integrate WCT functionality into other applications.  The ~wire-cell~ program provides a command line interface to ~Main~.  Likewise, the ~WCLS_tool~ in the ~larwirecell~ packages of LArSoft providse an /art/ / FHiCL interface to ~Main~.
+
+* WCT "apps"
+
+Finally, this package provides a number of simple WCT "apps" classes.
+Typically, one or more "app" instance is used via ~Main~ to provide some
+top-level execution.  Provided here are ~ConfigDumper~ and ~NodeDumper~
+which are more examples than useful.  See ~TbbFlow~ from the ~tbb~ sub
+package and ~Pgrapher~ from the ~pgraph~ package for the two most used
+apps.

apps/apps/wcsonnet.cxx

@@ -33,9 +33,11 @@ int main(int argc, char** argv)
 {
     CLI::App app{"wcsonnet is a Wire-Cell Toolkit aware Jsonnet compiler"};
 
-    std::string filename;
+    std::string filename, output="/dev/stdout";
     std::vector<std::string> load_path, extvars, extcode, tlavars, tlacode;
 
+    app.add_option("-o,--output", output,
+                   "Output file");
     app.add_option("-P,--path", load_path,
                    "Search paths to consider in addition to those in WIRECELL_PATH")->type_size(1)->allow_extra_args(false);
     app.add_option("-V,--ext-str", extvars,
@@ -86,7 +88,8 @@ int main(int argc, char** argv)
                            m_tlavars, m_tlacode);
 
     auto jdat = parser.load(filename);
-    std::cout << jdat << std::endl;
+    std::ofstream out(output);
+    out << jdat << std::endl;
 
     return 0;
 }

apps/apps/wire-cell.cxx

@@ -15,11 +15,10 @@ int main(int argc, char* argv[])
 
     try {
         rc = m.cmdline(argc, argv);
-        if (rc) {
-            return rc;
+        if (rc == 0) {
+            m.initialize();
+            m();
         }
-        m.initialize();
-        m();
     }
     catch (Exception& e) {
         cerr << errstr(e) << endl;

apps/inc/WireCellApps/Main.h

@@ -30,6 +30,10 @@ namespace WireCell {
         ///
         /// Or, one can use subsequent methods for more fine-grained
         /// setup and execution.
+        ///
+        /// Return code rc:
+        /// rc = 1 : if --help or --version,
+        /// rc = 0 : if normal running should commence
         int cmdline(int argc, char* argv[]);
 
         /// Individual setup methods called by cmdline() or called