Dev Guide updates

.github/workflows/build_workflow.yml

@@ -2,11 +2,15 @@ name: CI/CD Build Workflow
 
 on:
   push:
+    # When a branch is pushed to GitHub, run this workflow.
+    # This will show up as the checks to pass on a pull request.
     branches: [main]
 
   pull_request:
+    # When a pull request is merged, run this workflow.
     branches: [main]
 
+# These are the 3 jobs that show up under "All checks have passed" on GitHub.
 jobs:
   pre-commit-hooks:
     runs-on: ubuntu-latest
@@ -22,7 +26,9 @@ jobs:
 
       # Run all pre-commit hooks on all the files.
       # Getting only staged files can be tricky in case a new PR is opened
-      # since the action is run on a branch in detached head state
+      # since the action is run on a branch in detached head state.
+      # This is the equivalent of running "pre-commit run --all-files" locally.
+      # If you commit with the `--no-verify` flag, this check may fail.
       - name: Install and Run Pre-commit
         uses: pre-commit/[email protected]
 
@@ -65,6 +71,7 @@ jobs:
         run: |
           python -m unittest tests/test_*.py
 
+  # If the branch updates documentation, then the docs will need to be updated.
   publish-docs:
     if: github.event_name == 'push'
     needs: [pre-commit-hooks, build]
@@ -100,6 +107,7 @@ jobs:
           channel-priority: strict
           auto-update-conda: true
 
+      # sphinx-multiversion allows for version docs.
       - name: Build Sphinx Docs
         run: |
           cd docs

.github/workflows/release_workflow.yml

@@ -1,10 +1,15 @@
 name: CI/CD Release Workflow
 
 on:
+  # This is run on release, meaning it is run when these steps are followed:
+  # https://e3sm-project.github.io/zstash/_build/html/main/dev_guide/release.html#releasing-on-github-production-releases
+  # That means this workflow is not run for release candidates.
   release:
     types: [published]
 
 jobs:
+
+  # On a release, the docs should always be published.
   publish-docs:
     runs-on: ubuntu-latest
     defaults:
@@ -38,6 +43,7 @@ jobs:
           channel-priority: strict
           auto-update-conda: true
 
+      # sphinx-multiversion allows for version docs.
       - name: Build Sphinx Docs
         run: |
           cd docs

docs/source/dev_guide/ci.rst

@@ -0,0 +1,24 @@
+Understanding CI
+========================
+
+In this guide, we'll cover:
+
+* Build Workflow
+* Release Workflow
+* GitHub Actions
+
+Build Workflow
+--------------
+
+The build workflow is at https://github.com/E3SM-Project/zstash/blob/main/.github/workflows/build_workflow.yml. See comments in the file for in-depth explanations of each step.
+
+Release Workflow
+----------------
+
+The release workflow is at https://github.com/E3SM-Project/zstash/blob/main/.github/workflows/release_workflow.yml. See comments in the file for in-depth explanations of each step.
+
+GitHub Actions
+--------------
+Both of these workflows are run by GitHub Actions. See https://github.com/E3SM-Project/zstash/actions.
+
+When a pull request is made, the build workflow is run automatically on the pushed branch. When the pull request is merged, the build workflow is once again run, but this time on the ``main`` branch.

docs/source/dev_guide/index.rst

@@ -6,4 +6,6 @@ Developer Guide
    :maxdepth: 2
 
    project-standards
-   release
+   ci
+   pre_release_testing
+   release

docs/source/dev_guide/pre_release_testing.rst

@@ -0,0 +1,38 @@
+***************************************
+Testing directions for making a release
+***************************************
+
+1. Have three shells open: one on Chrysalis, one on Compy, and one on Perlmutter. Do the following steps on each machine.
+
+   * If running on Perlmutter, it is preferable to run from ``$CSCRATCH`` rather than ``/global/homes``. Running from the latter may result in a 'Resource temporarily unavailable' error.
+   * If running on Compy, it is necessary to run ``test_globus.py`` from a sub-directory of ``/compyfs`` rather than ``/qfs``. 
+
+2. ``cd`` to the ``zstash`` directory.
+
+3. Check out a branch to test on. ::
+
+     git fetch upstream main
+     git checkout -b test_pre_rc<#> upstream/main
+     git log # check the commits match https://github.com/E3SM-Project/zstash/commits/main
+
+4. Set up a new development environment. This ensures that testing will use the latest conda changes. Note that you will need to run ``conda remove -n zstash_dev_pre_rc<#> --all`` first if you have previously done this step. ::
+
+     mamba clean --all
+     mamba env create -f conda/dev.yml -n zstash_dev_pre_rc<#>
+     conda activate zstash_dev_pre_rc<#>
+     pip install .
+
+5. Activate Globus. Go to https://www.globus.org/. Log in with your NERSC credentials. Activate the following Globus endpoints using the corresponding credentials:
+
+   * `Chrysalis <https://app.globus.org/file-manager/collections/61f9954c-a4fa-11ea-8f07-0a21f750d19b/overview>`_
+   * `Compy <https://app.globus.org/file-manager/collections/68fbd2fa-83d7-11e9-8e63-029d279f7e24>`_
+   * `Perlmutter/NERSC <https://app.globus.org/file-manager/collections/9d6d99eb-6d04-11e5-ba46-22000b92c6ec>`_.
+
+6. Run the unit tests with ``python -m unittest tests/test_*.py``.
+
+   * If there are any failures, fix the code, and then go back to step 4.
+   * If there are no failures, proceed to the next step.
+
+7. Make a pull request and merge any changes. This keeps the repo updated with the latest testing configurations.
+
+8. Create the release candidate by following the directions at https://e3sm-project.github.io/zstash/_build/html/main/dev_guide/release.html.

docs/source/dev_guide/release.rst

@@ -11,20 +11,20 @@ In this guide, we'll cover:
 Bumping the Version
 -------------------
 
-1. Checkout the latest ``main``.
-2. Checkout a branch with the name of the version.
+1. Checkout a branch with the name of the version.
 
     ::
 
+        git fetch upstream main
         # Prepend "v" to <version>
         # For release candidates, append "rc" to <version>
-        git checkout -b v<version> -t origin/<version>
+        git checkout -b v<version> upstream/main
 
-3. Bump version using tbump.
+2. Bump version using tbump.
 
     ::
 
-        # Exclude "v" and <version> should match step 2
+        # Exclude "v" and <version> should match the above step.
         # --no-tag is required since tagging is handled in "Releasing on GitHub"
         $ tbump <version> --no-tag
 
@@ -44,7 +44,8 @@ Bumping the Version
         $ git push origin v1.2.0
         :: Looking good? (y/N)
         >
-4. Create a pull request to the main repo and merge it.
+
+3. Create a pull request to the main repo and merge it.
 
 .. _github-release:
 
@@ -57,8 +58,10 @@ Releasing on GitHub: release candidates
 
 	$ git checkout main
 	$ git fetch upstream
-	$ git rebase upstream/main
+	$ git reset --hard upstream/main
 	$ git tag -a v1.2.0rc1 -m "v1.2.0rc1"
+	# Delete the branch from the tbump step. Otherwise, the push command won't work.
+	$ git branch -D v1.2.0rc1
 	$ git push upstream v1.2.0rc1
 
 Releasing on GitHub: production releases
@@ -78,14 +81,35 @@ Releasing on GitHub: production releases
 Releasing on conda-forge: release candidates
 --------------------------------------------
 
-1. Make a PR to `conda-forge <https://github.com/conda-forge/zstash-feedstock/>`_ from your fork of the feedstock. Note that the conda-forge bot does not work for release candidates.
+1. If you don't have a local version of the conda-forge repo, run: ::
+
+     git clone [email protected]:conda-forge/zstash-feedstock.git
+     git remote add upstream [email protected]:conda-forge/zstash-feedstock.git
+
+2. If you don't have a fork of the conda-forge repo, on `conda-forge <https://github.com/conda-forge/zstash-feedstock/>`_, click the "Fork" button in the upper right hand corner. Then, on your fork, click the green "Code" button, and copy the SSH path. Run: ::
+
+     git remote add <your fork name> <SSH path for your fork>
+
+3. Get the sha256 of the tag you made in "Releasing on GitHub: release candidates": ::
+
+     curl -sL https://github.com/E3SM-Project/zstash/archive/v1.2.0rc1.tar.gz | openssl sha256
 
-   * Start from the current dev branch and update the version number and the sha256 sum manually.
-   * Set the build number back to 0 if needed.
-   * Make the dev branch the target of the PR. Then, the package build on conda-forge will end up with the ``zstash_dev`` label.
+4. Make changes on a local branch: ::
 
-2. Check the https://anaconda.org/conda-forge/zstash page to view the newly updated package. Release candidates are assigned the ``zstash_dev`` label.
+     $ git fetch upstream dev
+     $ git checkout -b v1.2.0rc1 upstream/dev # You can name the branch anything you want
+     # In `recipe/meta.yaml`, update the version and sha256 (and the build number if needed):
+     {% set version = "1.2.0rc1" %} # Set to your version
+     sha256: ... # The sha256 from the previous step
+     number: 0 # build > number should always be 0
+     $ git add -A
+     $ git commit -m "v1.2.0rc1"
+     $ git push <your fork name> v1.2.0rc1
 
+5. Note that the conda-forge bot does not work for release candidates. So, make a PR manually from your fork of the feedstock to the ``dev`` branch of `conda-forge <https://github.com/conda-forge/zstash-feedstock/>`_. Then, the package build on conda-forge will end up with the ``zstash_dev`` label. You can add the "automerge" label to have the PR automatically merge once CI checks pass.
+
+6. 6. After merging, CI runs again (in a slightly different way). Then, check the https://anaconda.org/conda-forge/zstash page to view the newly updated package. Release candidates are assigned the ``zstash_dev`` label. Note that it takes about 15 minutes for the files to propagate across conda-forge's mirroring services, which must happen before you can use the files.
+
 Releasing on conda-forge: production releases
 ------------------
 
@@ -102,3 +126,12 @@ Creating a New Version of the Documentation
 1. Be sure to have already completed :ref:`Releasing On GitHub <github-release>`. This triggers the CI/CD workflow that handles publishing documentation versions.
 2. Wait until the CI/CD build is successful. You can view all workflows at `All Workflows <https://github.com/E3SM-Project/zstash/actions>`_.
 3. Changes will be available on the `zstash documentation page <https://e3sm-project.github.io/zstash/>`_.
+
+Extra Resources
+---------------
+
+Conda-forge:
+
+* https://conda-forge.org/docs/user/introduction.html#why-conda-forge
+* https://conda-forge.org/docs/maintainer/infrastructure.html#admin-web-services
+* https://acme-climate.atlassian.net/wiki/spaces/IPD/pages/3616735236/Releasing+E3SM+Software+on+Anaconda+conda-forge+channel