Unify CLI docs at toplevel README.md (#38)

* Updated docs for running `PROBE`

* update tasks.md

* merge terminology section of probe_frontend README into glossary

* spelling fixes

* syntax fixes

* re-added translation joke (will revert upon request)

* fix whitespace

* Edits

---------

Co-authored-by: Samuel Grayson <[email protected]>

README.md

@@ -10,15 +10,17 @@ The provenance graph can help us re-execute the program, containerize the progra
 
 ## Reading list
 
-- [./docs/acm-rep-pres.pdf](./docs/acm-rep-pres.pdf) for an introduction to my work.
 - [Provenance for Computational Tasks: A Survey by Juliana Freire, David Koop, Emanuele Santos, and Cláudio T. Silva](https://sci.utah.edu/~csilva/papers/cise2008a.pdf) for an overview of provenance in general
 - [CDE: Using System Call Interposition to Automatically Create Portable Software Packages by Philip J. Guo and Dawson Engler](https://www.usenix.org/legacy/events/atc11/tech/final_files/GuoEngler.pdf) for a seminal system-level provenance tracer.
+- [Techniques for Preserving Scientific Software Executions: Preserve the Mess or Encourage Cleanliness?](https://curate.nd.edu/articles/journal_contribution/Techniques_for_Preserving_Scientific_Software_Executions_Preserve_the_Mess_or_Encourage_Cleanliness_/24824439?file=43664937) discusses whether enabling automatic-replay is actually a good idea. A cursory glance makes PROBE seem more like "preserving the mess", but I think, with some care in the design choices, it actually can be more like "encouraging cleanliness", for example, by having heuristics that help cull/simplify provenance and generating human readable/editable package-manager recipes.
+- [SoK: History is a Vast Early Warning System: Auditing the Provenance of System Intrusions](https://adambates.org/documents/Inam_Oakland23.pdf) discusses previous approaches for auditing, processing, and detecting intrusions, which is very related.
+- [System-Level Provenance Tracers `./docs/acm-rep-pres.pdf`](./docs/acm-rep-pres.pdf) for a motivation of this work.
 
-## Running
+## Installing PROBE
 
-1. Install Nix with flakes.
+1. Install Nix with flakes. This can be done on any Linux (including Ubuntu, RedHat, Arch Linux, not just NixOS), MacOS X, or even Windows Subsystem for Linux.
 
-   - If you don't already have Nix, use the [Determinate Systems installer](https://install.determinate.systems/).
+   - If you don't already have Nix on your system, use the [Determinate Systems installer](https://install.determinate.systems/).
 
    - If you already have Nix (but not NixOS), enable flakes by adding the following line to `~/.config/nix/nix.conf` or `/etc/nix/nix.conf`:
 
@@ -28,101 +30,72 @@ The provenance graph can help us re-execute the program, containerize the progra
 
    - If you already have Nix and are running NixOS, enable flakes with by adding `nix.settings.experimental-features = [ "nix-command" "flakes" ];` to your configuration.
 
-2. Acquire the source code: `git clone https://github.com/charmoniumQ/PROBE && cd PROBE`
-
-3. Run `nix develop` to enter the development environment.
-
-4. `cd probe_src`
-
-5. Run PROBE with `./PROBE record --make head ../flake.nix`
+2. Run `nix env -i github:charmoniumQ/PROBE#probe-bundled`.
 
-   - Note that `--make` will cause libprobe to be re-compiled or compiled, so if you make changes to libprobe or you haven't compiled it before, use this flag.
+3. Now you should be able to run `probe record [-f] [-o probe_log] <cmd...>`, e.g., `probe record ./script.py --foo bar.txt`. See below for more details.
 
-   - This will output `probe_log` unless you pass `--output <desired_output_path>`.
+4. To view the provenance, run `probe dump [-i probe_log]`. See below for more details.
 
-   - Also note `--debug` which runs a build of libprobe that is more verbose, has runtime checks, and has debug symbols.
+5. Run `probe --help` for more details.
 
-   - Also note `--gdb` which loads libprobe, your command, and drops you into a GDB shell.
+## What does `probe record` do?
 
-6. Try `./PROBE dump`, which will read `probe_log` unless you pass `--input <desired_input_path>`. This should show a series of provenance operations, e.g.,
-  
-   ```
-   InitExecEpochOp(process_id=116448, exec_epoch=0, process_birth_time=timespec(tv_sec=1718127174, tv_nsec=333602925), program_name='head')
-   InitThreadOp(process_id=116448, process_birth_time=timespec(tv_sec=1718127174, tv_nsec=333602925), exec_epoch=0, sams_thread_id=0)
-   OpenOp(path=Path(dirfd_minus_at_fdcwd=0, path='../flake.nix', device_major=0, device_minor=32, inode=123261484, stat_valid=True, dirfd_valid=True), flags=0, mode=0, fd=6, ferrno=0)
-   CloseOp(low_fd=6, high_fd=6, ferrno=0)
-   CloseOp(low_fd=1, high_fd=1, ferrno=0)
-   CloseOp(low_fd=2, high_fd=2, ferrno=0)
-   ```
+The simplest invocation of the `probe` cli is:
 
-## Glossary
+```bash
+probe record <CMD...>
+```
 
-- **Executable**: An executable is a file that contains the information telling a operating system and computer hardware how to execute a specific task. In paarticular, they instruct the operating system to load certain hardware instructions and data to certain locations in memory. In Linux, they are specified using the [ELF format](https://www.wikiwand.com/en/Elf_format).
+This will run `<CMD...>` under the benevolent supervision of libprobe, outputting the probe record to a temporary directory. Upon the process exiting, `probe` it will transcribe the record directory and write a probe log file named `probe_log` in the current directory.
 
-- **Loader**: A part of the operating system which loads the instructions and data specified by an executable file into memory. In Linux, the loader is called [`ld.so`](https://www.man7.org/linux/man-pages/man8/ld.so.8.html).
+If you run this again you'll notice it throws an error that the output file already exists, solve this by passing `-o <PATH>` to specify a new file to write the log to, or by passing `-f` to overwrite the previous log.
 
-- **Shared library**: A file containing a mapping from symbols to data or instructions. An executable can request a certain shared library be loaded at a certain place. The shared library can be specified by absolute path or by relative path, which will be resolved according to specific rules (see `RPATH` in [`man ld.so`](https://www.man7.org/linux/man-pages/man8/ld.so.8.html)).
+<!--
+This is stuff that normal users don't need to know about. Developers may find it useful:
 
-- **Libc**: A shared library that implements the functions defined in ANSI C. "In theory", the syscall table is the interface between an OS and an application; the OS provides `open`  at syscall `SYS_OPEN`. "In practice", libc is the interface between the OS and an applications; the OS's libc will define `FILE* fopen(const char* filename, const char* mode)`. This is because the syscall table has limited slots and expensive to invoke. Splitting paths is certainly OS dependent, but OS designers don't want to waste a valuable syscall slot for something that can be done in userspace, so they put it in libc. GNU Libc (aka glibc) is the most common implementation of Libc on Linux. The principle of using libc as an interface is quite strong; Rust uses satatic (not shared) libraries for everything *except* libc, because it is the best way to interface the OS. There are some notable exceptions, however; Go language does not use dynamically link against libc. Those programs are opaque to PROBE until future work, which may involve binary rewriting or ptrace.
+The transcription process can take some time (but usually no more than a few seconds unless disk IO is exceptionally slow) after the program exits, if you don't want to automatically transcribe the record, you can pass the `-n` flag, this will change the default output path from `probe_log` to `probe_record`, and will output a probe record directory that can be transcribed to a probe log later with the `PROBE transcribe` command, however the probe record format is not stable, users are strongly encouraged to have `PROBE record` automatically transcribe the record directory immediately after the process exits. If you do separate the transcription step from recording, then transcription **must** be done on the same machine with the exact same version of the cli (and other constraints, see the [section on serialization formats](https://github.com/charmoniumQ/PROBE/blob/main/probe_src/probe_frontend/README.md#serialization-formats) for more details).
+-->
 
-- **Symbol**: A symbol is a string used as the name of a global variable or function, optionally with version information appended to it, e.g., `fopen@@GLIBC_2.2.5`. Shared library exports symbols that can be referenced from an executable.
 
-- **Library interposition** is a technique that replaces a function symbol in a common library, like Libc, with a "proxy symbol". The proxy symbol will usually find and call the "true symbol" in the underlying library, but it may do arbitrary filtering, logging, or pre/post-processing as well. The proxy symbol is said to "hook" or "override" the original symbol. Library interposition is described more in [Curry's 1994 USENIX paper on library interposition in System V](https://www.usenix.org/conference/usenix-summer-1994-technical-conference/profiling-and-tracing-dynamic-library-usage), although it was probably known before that.
+`probe record` does **not** pass your command through a shell, any subshell or environment substitutions will still be performed by your shell before the arguments are passed to `probe`. But it won't understand flow control statements like `if` and `for`, shell builtins like `cd`, or shell aliases/functions.
 
-- **`$LD_PRELOAD`**: an environment variable that tells the loader to load the colon-delimited list of shared libraries _before_ loading the other shared libraries that the program requests. These libraries will get searched first when the program requests a certain symbol. One can use `$LD_PRELOAD` to  implement library interpositioning on Linux.
+If you need these you can either write a shell script and invoke `probe record` on that, or else run:
 
-- **Provenance**: the process by which a particular element of the system's global state (often a file) came to have its current value. The process often includes an executable, its arguments, its environments, the files it read, and the provenance of the executable and read-files. We can observe some provenance by tracing one process, but most of the benefits are conferred by tracing a _set_ of interacting processes, e.g., a shell script, Makefile, or workflow.
+```bash
+probe record bash -c '<SHELL_CODE>'
+```
 
-- **Provenance operation** (our term, also called "prov op"): an operation that reads or writes global state. Often the global state is a file in the filesystem, but it can also be calling for the current time, calling for an OS-level pseudo-random number (i.e., `getrand`), forking a process, or waiting on a process. If we observe all provenance operations relating to a specific element, we can infer the provenance of that element.
+(any flag after the first positional argument is treated as an argument to the command, not `probe`).
 
-- **libprobe** (our program): uses `$LD_PRELOAD` to hook library calls that invoke prov ops (see the implementation at [`./probe_src/libprobe`](./probe_src/libprobe)). It saves a record specifying the input or output for later analysis. Depending on a runtime option, it will save the original file contents (slow but replayable) or just the name of the orginal file (fast but not replayable). Libprobe uses a [memory-mapped](https://www.wikiwand.com/en/Memory-mapped_file) [arena allocator](https://www.wikiwand.com/en/Region-based_memory_management) to log records to disk at high speeds (see the implementation at `[./probe_src/arena/README.md`](./probe_src/arena/README.md)).
+If you get tired of typing `probe record ...` in front of every command you wish to record, consider recording your entire shell session:
 
-- **Exec epoch** (our term): the [exec-family](https://www.man7.org/linux/man-pages/man3/exec.3.html) of syscalls replace the _current_ process by loading a new one. The period in between subsequent execs or between an exec and an exit is called an "exec epoch". Note that we consider the thread's lifetime to be a sub-interval of the exec epoch (each exec epochs contains threads), since a call to `exec` kills all threads (Linux considers the main thread as killed and re-spawned at the exec boundary, even though it has the same PID and TID).
+``` bash
+$ probe record bash
+bash$ ls -l
+bash$ # do other commands
+bash$ exit
 
-- **PROBE log** (our output): A tar archive of logs for each process, for each exec epoch, for each thread spawned during that exec epoch. Each log contains an ordered list of prov ops.
+$ probe dump
+<dumps history for entire bash session> 
+```
 
-- **Program order** (our adaptation of an existing term): is a [partial order](https://www.wikiwand.com/en/Partially_ordered_set) on the set of prov ops, which is the order the operations appear in a dynamic trace of a single thread.
+## What can I do with provenance?
 
-- **Process fork-join order** (our term): is a partial order on the set of prov ops, which is that if A forks B, then the `fork` prov op in A should precede the first prov op of B. Likewise if A calls `waitpid` and `waitpid` returns the PID of B, then the last prov op in B should precede completion of the prov op `waitpid` in A.
+That's a huge [work in progress](https://github.com/charmoniumQ/PROBE/pulls).
 
-- **Happens-before order** (our term of an [existing term](https://www.wikiwand.com/en/Happens-before)): is the union of program order and process fork-join order.
-  - If a write from global state happens-before a read to global state, then the information *may* flow from the write to the read.
-  - If a read happens-before a write, information *can't* flow from the write to the read.
-  - If neither is true, information *may* flow.
+## Developing PROBE
 
-- **Prov op interval** (our term): is an pair of prov ups, one called "upper" the other called "lower", where the upper precedes the lower in happens-before order. We say "x is in the interval" for a prov op x, if the upper prov op happens-before x and x happens-before the lower prov op.
+1. Follow the previous step to install Nix.
 
-- **Open/close interval** (our term): is a prov op interval between the `open` of a file and its associated `close`. We think it would be too expensive to track individual file reads and writes. Therefore, we track the opens and closes instead. Any prov op in that interval may read or write that file, depending on the open mode. Although we lose specificity by tracking open/close intervals instead of reads/writes, our hypothesis is that correct programs will often separate their reads from their writes by process fork-join order.
-
-  E.g., suppose `foo` and `bar` are some black-box executables. We don't know how to interpret their shell arguments. Suppose they are composed in a shell script like `foo && bar`, which we run with PROBE. We would get the following PROBE log (very roughly):
-
-  - PID 100:
-    - Exec `bash`
-    - Fork (returns 101)
-    - Waitpid 101
-    - Fork (returns 102)
-    - Waitpid 102
-  - PID 101:
-    - Exec `foo`
-    - Open `input_file` for reading
-    - Open `temp_file` for writing
-    - (many reads and writes happen, which we do not track)
-    - Close all files
-  - PID 102:
-    - Exec `bar`
-    - Open `output_file` for writing
-    - Open `temp_file` for reading
-    - (many reads and writes happen, which we do not track)
-    - Close all files
+2. Acquire the source code: `git clone https://github.com/charmoniumQ/PROBE && cd PROBE`
 
-  The entire close of `temp_file` in PID 101 precedes waitpid 101 in fork-join order, waitpid 101 precedes fork 102 in program order, and fork 102 precedess the open of `temp_file` in PID 102, so we can conclude that information may flow from `foo` to `bar`.
+3. Run `nix develop`. This will leave you in a _Nix development shell_, with all the development tools you need to develop and build PROBE. It is like a virtualenv, in that it is isolated from your system's pre-existing tools. In the development shell, we all have the same version of Python with all the same packages. You can exit it by dyping `exit`.
 
-  From which, we can deduce the provenance graph `input_file` -> `foo` -> `temp_file` -> `bar` -> `output_file` with only the open/close intervals and happens-before order.
+4. From _within the development shell_, type `just compile`. This compiles the Rust, C, and generated-Python components. If you hack on either, run `just compile` again before continuing.
 
-## FAQ
+5. The manually-written Python scripts should already be added to the `$PYTHONPATH`. You should be able to edit them in place.
 
-- Why doesn't your flake define a Nix app or Nix package?
-  - Because I have a finite amount of time, and I'm still in the unstable development phase.
+6. Run `probe <args...>` or `python -m probe_py.manual.cli <args...>` to invoke the Rust or Python code respectively.
 
 ## Prior art
 

flake.nix

@@ -71,7 +71,7 @@
               mkdir $out $out/bin
               makeWrapper \
                 ${self.packages.${system}.probe-cli}/bin/probe \
-                $out/bin/PROBE \
+                $out/bin/probe \
                 --set __PROBE_LIB ${libprobe}/lib
             '';
           };