Merge pull request #224 from lsst-sqre/tickets/DM-44137

DM-44137: Add extend_excludes_for_non_index_source
lsst-sqre · Apr 30, 2024 · 1374087 · 1374087
2 parents 0959a15 + e2b6b98
commit 1374087
Show file tree

Hide file tree

Showing 11 changed files with 280 additions and 7 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.5.0
+    rev: v4.6.0
     hooks:
       - id: check-yaml
         exclude: ^src/documenteer/storage/localtemplates/technote/ci\.yaml
@@ -22,12 +22,12 @@ repos:
           - toml
 
   - repo: https://github.com/psf/black
-    rev: 24.3.0
+    rev: 24.4.2
     hooks:
       - id: black
 
   - repo: https://github.com/asottile/blacken-docs
-    rev: 1.14.0
+    rev: 1.16.0
     hooks:
       - id: blacken-docs
         additional_dependencies: [black==22.12.0]

diff --git a/changelog.d/20240430_155714_jsick_DM_44137.md b/changelog.d/20240430_155714_jsick_DM_44137.md
@@ -0,0 +1,11 @@
+### Bug fixes
+
+- Technotes ignore files in the repository with `.md`, `.rst`, and `.ipynb` extensions if they aren't the index file. Since technotes are single-page documents, only the index file should be used as a source file. This change lets authors include auxiliary notebooks with their technotes without having to explicitly exclude them from the technote build process. This is implemented with `technote.conf.extend_excludes_for_non_index_source`.
+
+- In `documenteer.ext.lssttasks`, attempt to import `_pseudo_parse_arglist` from `sphinx.domains.python._annotations` before falling back to the `sphinx.domains.python` module. Ultimately this is a workaround.
+
+- Fix setting the rebuild condition for the `documenteer.ext.githubbibcache` extension.
+
+### Other changes
+
+- Added `defusedxml` as a dev dependency. This is used by Sphinx's `sphinx.testing.fixtures`, but isn't included as a dependency by Sphinx itself. This change ensures that `defusedxml` is installed when running the tests.
diff --git a/demo/ipynb-technote/extra-notebook.ipynb b/demo/ipynb-technote/extra-notebook.ipynb
@@ -0,0 +1,97 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Demo technote\n",
+    "\n",
+    "```{abstract}\n",
+    "A *technote* is a web-native single page document that enables rapid technical communication within and across teams.\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Introduction\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "A parenthetical citation {cite:p}`SQR-083`. And a textual citation {cite:t}`SQR-083`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Method\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "A list:\n",
+    "\n",
+    "- First item\n",
+    "- Second item\n",
+    "- Third item\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"Hello world\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Analysis\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "```{prompt} bash\n",
+    "git add index.rst\n",
+    "```\n",
+    "\n",
+    "Some following text.\n",
+    "\n",
+    "```{mermaid}\n",
+    "graph TD\n",
+    "    A[Square Rect] -- Link text --> B((Circle))\n",
+    "    A --> C(Round Rect)\n",
+    "    B --> D{Rhombus}\n",
+    "    C --> D\n",
+    "```\n",
+    "\n",
+    "```{diagrams} diagram.py\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## References\n",
+    "\n",
+    "```{bibliography}\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/demo/ipynb-technote/subdir/subdir-notebook.ipynb b/demo/ipynb-technote/subdir/subdir-notebook.ipynb
@@ -0,0 +1,97 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Demo technote\n",
+    "\n",
+    "```{abstract}\n",
+    "A *technote* is a web-native single page document that enables rapid technical communication within and across teams.\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Introduction\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "A parenthetical citation {cite:p}`SQR-083`. And a textual citation {cite:t}`SQR-083`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Method\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "A list:\n",
+    "\n",
+    "- First item\n",
+    "- Second item\n",
+    "- Third item\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"Hello world\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Analysis\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "```{prompt} bash\n",
+    "git add index.rst\n",
+    "```\n",
+    "\n",
+    "Some following text.\n",
+    "\n",
+    "```{mermaid}\n",
+    "graph TD\n",
+    "    A[Square Rect] -- Link text --> B((Circle))\n",
+    "    A --> C(Round Rect)\n",
+    "    B --> D{Rhombus}\n",
+    "    C --> D\n",
+    "```\n",
+    "\n",
+    "```{diagrams} diagram.py\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## References\n",
+    "\n",
+    "```{bibliography}\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/pyproject.toml b/pyproject.toml
@@ -47,6 +47,7 @@ dev = [
     # Test depedendencies for analyzing HTML output
     "lxml",
     "cssselect",
+    "defusedxml",  # used by Sphinx but not declared as a dependency
     # Extensions for documenteer's own docs
     "sphinx-click",
     "sphinxcontrib-autoprogram",
@@ -56,6 +57,7 @@ dev = [
     "types-PyYAML",
     "types-docutils",
     "types-mock",
+    # Testing dependencies
 ]
 guide = [
     # Theme and extensions for Rubin user guide projects

diff --git a/src/documenteer/conf/__init__.py b/src/documenteer/conf/__init__.py
@@ -1,5 +1,6 @@
 from ._toml import DocumenteerConfig
 from ._utils import (
+    extend_excludes_for_non_index_source,
     extend_static_paths_with_asset_extension,
     get_asset_path,
     get_template_dir,
@@ -8,6 +9,7 @@
 __all__ = [
     "get_asset_path",
     "extend_static_paths_with_asset_extension",
+    "extend_excludes_for_non_index_source",
     "get_template_dir",
     "DocumenteerConfig",
 ]
diff --git a/src/documenteer/conf/_utils.py b/src/documenteer/conf/_utils.py
@@ -113,3 +113,37 @@ def working_tree_dir(self) -> Path:
         if path is None:
             raise RuntimeError("Git repository is not available.")
         return Path(path)
+
+
+def extend_excludes_for_non_index_source(
+    exclude_patterns: list[str],
+    extension: str,
+    working_dir: Path | None = None,
+) -> None:
+    """Extend the exclude_patterns configuration to include files with a
+    specific extension that aren't the index file.
+
+    This is useful for technotes where only the ``index{.ext}`` file is valid
+    as a source file.
+
+    Parameters
+    ----------
+    exclude_patterns
+        The Sphinx configuration, ``exclude_patterns``. This configuration is
+        modified in place.
+    extension
+        The file extension to exclude. Examples: ``"ipynb"``, ``"md"``,
+        ``"rst"``. The extension does not include the leading period.
+    working_dir
+        The working directory to search for files. If not provided, the
+        current working directory is used. This should be equivalent to the
+        directory where the Sphinx configuration is located.
+    """
+    if working_dir is None:
+        cwd = Path.cwd()
+    else:
+        cwd = working_dir
+
+    for p in cwd.glob(f"**/*.{extension}"):
+        if p.name != f"index.{extension}":
+            exclude_patterns.append(str(p.relative_to(cwd)))
diff --git a/src/documenteer/conf/technote.py b/src/documenteer/conf/technote.py
@@ -5,6 +5,7 @@
 from technote.sphinxconf import *  # noqa: F401 F403
 
 from documenteer.conf import (
+    extend_excludes_for_non_index_source,
     extend_static_paths_with_asset_extension,
     get_asset_path,
     get_template_dir,
@@ -74,6 +75,11 @@
     "logo_alt_text": "Rubin Observatory logo",
 }
 
+# Exclude non-index.ipynb Jupyter Notebooks
+extend_excludes_for_non_index_source(exclude_patterns, "ipynb")  # noqa: F405
+extend_excludes_for_non_index_source(exclude_patterns, "md")  # noqa: F405
+extend_excludes_for_non_index_source(exclude_patterns, "rst")  # noqa: F405
+
 # Configure bibliography with the bib cache
 documenteer_bibfile_cache_dir = ".technote/bibfiles"
 documenteer_bibfile_github_repos = [

diff --git a/src/documenteer/ext/githubbibcache.py b/src/documenteer/ext/githubbibcache.py
@@ -116,11 +116,11 @@ def cache_bibfiles(app: Sphinx, config: Config) -> None:
 def setup(app: Sphinx) -> dict[str, Any]:
     """Set up the ``documenteer.ext.autocppapi`` Sphinx extensions."""
     # Configuration values
-    app.add_config_value("documenteer_bibfile_github_repos", "", True, [list])
+    app.add_config_value("documenteer_bibfile_github_repos", "", "env", [list])
     app.add_config_value(
         "documenteer_bibfile_cache_dir",
         Path("_build/bibfile-cache"),
-        True,
+        "env",
         [str, Path],
     )
     app.connect("config-inited", cache_bibfiles)

diff --git a/src/documenteer/ext/lssttasks/pyapisummary.py b/src/documenteer/ext/lssttasks/pyapisummary.py
@@ -13,7 +13,12 @@
     desc_signature,
     seealso,
 )
-from sphinx.domains.python import PyXRefRole, _pseudo_parse_arglist
+
+try:
+    from sphinx.domains.python._annotations import _pseudo_parse_arglist
+except ImportError:
+    from sphinx.domains.python import _pseudo_parse_arglist  # type: ignore
+from sphinx.domains.python import PyXRefRole
 from sphinx.errors import SphinxError
 from sphinx.util.inspect import signature as make_signature
 from sphinx.util.inspect import stringify_signature

diff --git a/tests/test_conf_utils.py b/tests/test_conf_utils.py
@@ -2,10 +2,16 @@
 
 from __future__ import annotations
 
+from pathlib import Path
+
 import pytest
 from sphinx.errors import ConfigError
 
-from documenteer.conf import get_asset_path, get_template_dir
+from documenteer.conf import (
+    extend_excludes_for_non_index_source,
+    get_asset_path,
+    get_template_dir,
+)
 
 
 def test_get_asset_path() -> None:
@@ -24,3 +30,16 @@ def test_get_template_dir() -> None:
     # This template dir doesn't exist
     with pytest.raises(ConfigError):
         get_asset_path("not-a-theme")
+
+
+def test_extend_excludes_for_non_index_source() -> None:
+    ipynb_demo_dir = Path(__file__).parent.joinpath("../demo/ipynb-technote")
+
+    # Initial excludes
+    excludes = ["_build", "_static", "_templates", "conf.py", "README.rst"]
+
+    extend_excludes_for_non_index_source(excludes, "ipynb", ipynb_demo_dir)
+
+    assert "subdir/subdir-notebook.ipynb" in excludes
+    assert "extra-notebook.ipynb" in excludes
+    assert "index.ipynb" not in excludes