Merge pull request #312 from lsst/tickets/DM-49039

DM-49039: Change schema directory to lsst.sdm.schemas
lsst · Feb 27, 2025 · 70c4db0 · 70c4db0
2 parents 463462e + 2d4b2ab
commit 70c4db0
Show file tree

Hide file tree

Showing 43 changed files with 96 additions and 23 deletions.
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -1,6 +1,6 @@
 ## Checklist
 
-When making changes to YAML files in the [schemas](/lsst/sdm_schemas/blob/main/python/lsst/sdm_schemas/schemas) directory:
+When making changes to YAML files in the [schemas](/lsst/sdm_schemas/blob/main/python/lsst/sdm/schemas) directory:
 
 - [ ] If applicable, incremented the schema version number, following the guidelines in the [contribution guide](/lsst/sdm_schemas/blob/main/CONTRIBUTING.md)
 - [ ] Referred to the [documentation on specific schemas](/lsst/sdm_schemas/blob/main/CONTRIBUTING.md#specific-schema-documentation) for additional versioning information, change constraints, or tasks that may need to be performed, based on which schema is being updated
diff --git a/.github/workflows/compare.yaml b/.github/workflows/compare.yaml
@@ -28,7 +28,7 @@ jobs:
 
       - name: Set relative path to schema directory
         run: |
-          echo "SCHEMA_DIR=python/lsst/sdm_schemas/schemas" >> $GITHUB_ENV
+          echo "SCHEMA_DIR=python/lsst/sdm/schemas" >> $GITHUB_ENV
 
       - name: Get list of changed YAML files
         working-directory: current-ref

diff --git a/.gitignore b/.gitignore
@@ -14,7 +14,7 @@ python/lsst_sdm_schemas.egg-info
 __pycache__
 
 # LSST version
-python/lsst/sdm_schemas/version.py
+python/lsst/sdm/schemas/version.py
 
 # SCons build
 config.log

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -30,7 +30,7 @@ Aside from special cases, a pull request will have to pass all of these checks b
 These checks will:
 
 - [Build TAP_SCHEMA and DataLink resources](.github/workflows/build.yaml)
-- [Compare Schemas for Changes](.github/workflows/compare.yaml) - Schemas that are in the [deployed list](./python/lsst/sdm_schemas/schemas/deployed-schemas.txt) will fail this check.
+- [Compare Schemas for Changes](.github/workflows/compare.yaml) - Schemas that are in the [deployed list](./python/lsst/sdm/schemas/deployed-schemas.txt) will fail this check.
 - [Build the Schema Browser website](.github/workflows/docs.yaml)
 - [Check that `main` is not merged into the branch](.github/workflows/rebase_checker.yaml)
 - [Test that the schemas can be used to create databases](.github/workflows/test_databases.yaml)
@@ -42,7 +42,7 @@ The checks will rerun anytime the PR branch is updated.
 Schema Versioning
 -----------------
 
-Individual schemas may have their own internal [version](https://felis.lsst.io/user-guide/model.html#schema-version) for tracking changes, which is defined at the top of the file, as in [apdb.yaml](./python/lsst/sdm_schemas/schemas/apdb.yaml).
+Individual schemas may have their own internal [version](https://felis.lsst.io/user-guide/model.html#schema-version) for tracking changes, which is defined at the top of the file, as in [apdb.yaml](./python/lsst/sdm/schemas/apdb.yaml).
 (This is distinct from tags or versions of sdm_schemas itself, and not all schemas may be using them.)
 This version may need to be updated when making changes.
 

diff --git a/README.md b/README.md
@@ -5,41 +5,41 @@ This repository stores schema definitions for user-facing data
 products comprising the Science Data Model (SDM) of the
 [Rubin Observatory](https://rubinobservatory.org/).
 These schemas are defined in YAML files designed to be read with
-[Felis](https://github.com/lsst-dm/felis), a tool which can convert this data
+[Felis](https://github.com/lsst/felis), a tool which can convert this data
 to derived formats including SQL DDL and
 [TAP_SCHEMA](https://www.ivoa.net/documents/TAP/20180830/PR-TAP-1.1-20180830.html#tth_sEc4).
 These schema definitions serve as the "source of truth" for the observatory's core data models.
 
 Schemas
 -------
 
-The following schemas are maintained in this repository under the [schemas](./python/lsst/sdm_schemas/schemas) directory:
+The following schemas are maintained in this repository under the [schemas](./python/lsst/sdm/schemas) directory:
 
- * [imsim](./python/lsst/sdm_schemas/schemas/imsim.yaml) describes the outputs
+ * [imsim](./python/lsst/sdm/schemas/imsim.yaml) describes the outputs
    of the pipelines for LSSTCam-imSim, used to generate the data preview
    schemas, which are fixed at the time of their release, while `imsim` will
    continue to evolve. This schema is used by
    [ci_imsim](https://github.com/lsst/ci_imsim) to verify that the tabular data
    of the pipeline output files is conformant with the schema definition.
 
  * The various Data Preview (DP) schemas such as
-   [dp02_dc2](./python/lsst/sdm_schemas/schemas/dp02_dc2.yaml) represent
+   [dp02_dc2](./python/lsst/sdm/schemas/dp02_dc2.yaml) represent
    content that is being actively served by the various data previews.
    These are created from `imsim` at a specific point in time.
 
- * [apdb](./python/lsst/sdm_schemas/schemas/apdb.yaml) describes the schema
+ * [apdb](./python/lsst/sdm/schemas/apdb.yaml) describes the schema
    of the Alert Production Database (APDB) used for Alert Production with
    `ap_pipe` and for nightly `ap_verify` runs within continuous integration.
    Previous processing runs may differ from the current schema. The
    user-queryable Prompt Products Database (PPDB) is expected to have a very similar schema to the APDB.
 
- * [hsc](./python/lsst/sdm_schemas/schemas/hsc.yaml) describes the outputs of
+ * [hsc](./python/lsst/sdm/schemas/hsc.yaml) describes the outputs of
    the latest data release production pipelines for HyperSuprimeCam. This
    schema is used by [ci_hsc](https://github.com/lsst/ci_hsc) for verification
    of its output files.
 
 * The various `cdb` schemas such as
-  [cdb_latiss](./python/lsst/sdm_schemas/schemas/cdb_latiss.yaml) describe the
+  [cdb_latiss](./python/lsst/sdm/schemas/cdb_latiss.yaml) describe the
   data model of the [Consolidated Database](https://github.com/lsst-dm/consdb)
   or ConsDB, an image metadata database containing summarizations of
   Engineering Facilities Database (EFD) telemetry by exposure and visit time windows.

diff --git a/SConstruct b/SConstruct
@@ -5,5 +5,5 @@ scripts.BasicSConstruct(
     "sdm_schemas",
     disableCc=True,
     noCfgFile=True,
-    versionModuleName="python/lsst/sdm_schemas/version.py"
+    versionModuleName=None
 )
diff --git a/docs/APDB.md b/docs/APDB.md
@@ -1,13 +1,13 @@
 Alert Production Database (APDB) Schema
 =======================================
 
-The [apdb](../python/lsst/sdm_schemas/schemas/apdb.yaml) schema describes the Alert Production Database (APDB), which contains the results of processing from the [Alert Production Pipeline](https://github.com/lsst/ap_pipe).
+The [apdb](../python/lsst/sdm/schemas/apdb.yaml) schema describes the Alert Production Database (APDB), which contains the results of processing from the [Alert Production Pipeline](https://github.com/lsst/ap_pipe).
 
 Schema Versioning
 -----------------
 
 The versioning system of the APDB is outlined in [DMTN-269](https://dmtn-269.lsst.io/).
-The deployed database schema is a product of the [schema in sdm_schemas](../python/lsst/sdm_schemas/schemas/apdb.yaml) with additional processing by the APDB client library from the [dax_apdb](https://github.com/lsst/dax_apdb) repository.
+The deployed database schema is a product of the [schema in sdm_schemas](../python/lsst/sdm/schemas/apdb.yaml) with additional processing by the APDB client library from the [dax_apdb](https://github.com/lsst/dax_apdb) repository.
 Compatibility of the client executable with the actual database schema is determined by the version number in the YAML file and the version stored in the database metadata.
 
 Backward compatibility in the context of the APDB implies that a client using a schema with a higher minor version will be able to read and write into a database with a lower minor version -- in other words, an existing database instance need not be migrated to match the newer client interface.

diff --git a/docs/DRP.md b/docs/DRP.md
@@ -8,14 +8,14 @@ When new major data release productions occur, e.g., a new Data Preview or Data
 HSC Schema
 ----------
 
-The [HSC schema](../python/lsst/sdm_schemas/schemas/hsc.yaml) maps to the live pipelines as configured for the Subaru Hypersuprime-Cam instrument and its Strategic Survey Program, one of the primary precursor datasets used for LSST development.
+The [HSC schema](../python/lsst/sdm/schemas/hsc.yaml) maps to the live pipelines as configured for the Subaru Hypersuprime-Cam instrument and its Strategic Survey Program, one of the primary precursor datasets used for LSST development.
 The [ci_hsc_gen3](/lsst/ci_hsc_gen3) package is run in nightly Jenkins tests, as well as optionally prior to other pipeline code merges, and checks that the HSC schema matches the Parquet datasets produced by the pipeline definition at [`drp_pipe/pipelines/HSC/DRP-ci_hsc.yaml`](/lsst/drp_pipe/blob/main/pipelines/HSC/DRP-ci_hsc.yaml).
 The other HSC pipelines in [drp_pipe](/lsst/drp_pipe) should produce files with the same schemas as well, because they share almost all configuration with the [ci_hsc](/lsst/ci_hsc) pipeline.
 
 ImSim Schema
 ------------
 
-The [ImSim schema](../python/lsst/sdm_schemas/schemas/imsim.yaml) similarly maps to the live pipelines as configured for the LSST ImSim simulator, in particular as run for the [LSST Dark Energy Science Collaboration](https://lsstdesc.org/)'s "Data Challenge 2" project ([DESC DC2](https://dp0-2.lsst.io/)).
+The [ImSim schema](../python/lsst/sdm/schemas/imsim.yaml) similarly maps to the live pipelines as configured for the LSST ImSim simulator, in particular as run for the [LSST Dark Energy Science Collaboration](https://lsstdesc.org/)'s "Data Challenge 2" project ([DESC DC2](https://dp0-2.lsst.io/)).
   This is the same simulated dataset used for LSST's Data Preview 0.1 and 0.2, but the pipelines have evolved considerably since those productions.
 The [ci_imsim](/lsst/ci_imsim) package is run nightly in Jenkins, as well as optionally prior to other pipeline code merges, and checks that the ImSim schema match the Parquet datasets produced by the pipeline definition at [`drp_pipe/pipelines/LSSTCam-imSim/DRP-ci_imsim.yaml`](https://github.com/lsst/drp_pipe/blob/main/pipelines/LSSTCam-imSim/DRP-ci_imsim.yaml).
   The other `LSSTCam-imSim` pipelines in `drp_pipe` should produce files with the same schemas as well, because they share almost all configuration with the `ci_imsim` pipeline.

diff --git a/pyproject.toml b/pyproject.toml
@@ -35,9 +35,6 @@ dynamic = ["version"]
 [tool.setuptools.dynamic]
 version = { attr = "lsst_versions.get_lsst_version" }
 
-[tool.lsst_versions]
-write_to = "python/lsst/sdm_schemas/version.py"
-
 [project.urls]
 Homepage = "https://sdm-schemas.lsst.io"
 Source = "https://github.com/lsst/sdm_schemas"
@@ -55,4 +52,4 @@ license-files = ["COPYRIGHT", "LICENSE"]
 where = ["python"]
 
 [tool.setuptools.package-data]
-"lsst.sdm_schemas" = ["schemas/*.yaml"]
+"lsst.sdm.schemas" = ["**/*.yaml"]
diff --git a/python/lsst/__init__.py b/python/lsst/__init__.py
@@ -0,0 +1,24 @@
+# This file is part of sdm_schemas.
+#
+# Developed for the LSST Data Management System.
+# This product includes software developed by the LSST Project
+# (https://www.lsst.org).
+# See the COPYRIGHT file at the top-level directory of this distribution
+# for details of code ownership.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import pkgutil
+
+__path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/python/lsst/sdm/__init__.py b/python/lsst/sdm/__init__.py
@@ -0,0 +1,24 @@
+# This file is part of sdm_schemas.
+#
+# Developed for the LSST Data Management System.
+# This product includes software developed by the LSST Project
+# (https://www.lsst.org).
+# See the COPYRIGHT file at the top-level directory of this distribution
+# for details of code ownership.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import pkgutil
+
+__path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/python/lsst/sdm/schemas/__init__.py b/python/lsst/sdm/schemas/__init__.py
@@ -0,0 +1,28 @@
+# This file is part of sdm_schemas.
+#
+# Developed for the LSST Data Management System.
+# This product includes software developed by the LSST Project
+# (https://www.lsst.org).
+# See the COPYRIGHT file at the top-level directory of this distribution
+# for details of code ownership.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("lsst-sdm-schemas")
+except PackageNotFoundError:
+    # Package not installed or scons not run.
+    __version__ = "0.0.0"
diff --git a/python/lsst/sdm_schemas/schemas/apdb.yaml → python/lsst/sdm/schemas/apdb.yaml b/python/lsst/sdm_schemas/schemas/apdb.yaml → python/lsst/sdm/schemas/apdb.yaml
diff --git a/...sst/sdm_schemas/schemas/archive/README.md → python/lsst/sdm/schemas/archive/README.md b/...sst/sdm_schemas/schemas/archive/README.md → python/lsst/sdm/schemas/archive/README.md
diff --git a/...hemas/schemas/archive/baselineSchema.yaml → ...t/sdm/schemas/archive/baselineSchema.yaml b/...hemas/schemas/archive/baselineSchema.yaml → ...t/sdm/schemas/archive/baselineSchema.yaml
diff --git a/...sdm_schemas/schemas/archive/dp01_dc2.yaml → ...on/lsst/sdm/schemas/archive/dp01_dc2.yaml b/...sdm_schemas/schemas/archive/dp01_dc2.yaml → ...on/lsst/sdm/schemas/archive/dp01_dc2.yaml
diff --git a/...emas/schemas/archive/sdss_stripe82_01.yml → .../sdm/schemas/archive/sdss_stripe82_01.yml b/...emas/schemas/archive/sdss_stripe82_01.yml → .../sdm/schemas/archive/sdss_stripe82_01.yml
diff --git a/...t/sdm_schemas/schemas/archive/wise_00.yml → python/lsst/sdm/schemas/archive/wise_00.yml b/...t/sdm_schemas/schemas/archive/wise_00.yml → python/lsst/sdm/schemas/archive/wise_00.yml
diff --git a/...t/sdm_schemas/schemas/archive/wise_01.yml → python/lsst/sdm/schemas/archive/wise_01.yml b/...t/sdm_schemas/schemas/archive/wise_01.yml → python/lsst/sdm/schemas/archive/wise_01.yml
diff --git a/.../lsst/sdm_schemas/schemas/cdb_latiss.yaml → python/lsst/sdm/schemas/cdb_latiss.yaml b/.../lsst/sdm_schemas/schemas/cdb_latiss.yaml → python/lsst/sdm/schemas/cdb_latiss.yaml
diff --git a/...t/sdm_schemas/schemas/cdb_lsstcomcam.yaml → python/lsst/sdm/schemas/cdb_lsstcomcam.yaml b/...t/sdm_schemas/schemas/cdb_lsstcomcam.yaml → python/lsst/sdm/schemas/cdb_lsstcomcam.yaml
diff --git a/...dm_schemas/schemas/cdb_lsstcomcamsim.yaml → ...n/lsst/sdm/schemas/cdb_lsstcomcamsim.yaml b/...dm_schemas/schemas/cdb_lsstcomcamsim.yaml → ...n/lsst/sdm/schemas/cdb_lsstcomcamsim.yaml
diff --git a/..._schemas/schemas/cdb_startrackerfast.yaml → ...lsst/sdm/schemas/cdb_startrackerfast.yaml b/..._schemas/schemas/cdb_startrackerfast.yaml → ...lsst/sdm/schemas/cdb_startrackerfast.yaml
diff --git a/...chemas/schemas/cdb_startrackernarrow.yaml → ...st/sdm/schemas/cdb_startrackernarrow.yaml b/...chemas/schemas/cdb_startrackernarrow.yaml → ...st/sdm/schemas/cdb_startrackernarrow.yaml
diff --git a/..._schemas/schemas/cdb_startrackerwide.yaml → ...lsst/sdm/schemas/cdb_startrackerwide.yaml b/..._schemas/schemas/cdb_startrackerwide.yaml → ...lsst/sdm/schemas/cdb_startrackerwide.yaml
diff --git a/.../sdm_schemas/schemas/deployed-schemas.txt → python/lsst/sdm/schemas/deployed-schemas.txt b/.../sdm_schemas/schemas/deployed-schemas.txt → python/lsst/sdm/schemas/deployed-schemas.txt
diff --git a/...on/lsst/sdm_schemas/schemas/dp02_dc2.yaml → python/lsst/sdm/schemas/dp02_dc2.yaml b/...on/lsst/sdm_schemas/schemas/dp02_dc2.yaml → python/lsst/sdm/schemas/dp02_dc2.yaml
diff --git a/...sst/sdm_schemas/schemas/dp02_obscore.yaml → python/lsst/sdm/schemas/dp02_obscore.yaml b/...sst/sdm_schemas/schemas/dp02_obscore.yaml → python/lsst/sdm/schemas/dp02_obscore.yaml
diff --git a/...n/lsst/sdm_schemas/schemas/dp03_10yr.yaml → python/lsst/sdm/schemas/dp03_10yr.yaml b/...n/lsst/sdm_schemas/schemas/dp03_10yr.yaml → python/lsst/sdm/schemas/dp03_10yr.yaml
diff --git a/...on/lsst/sdm_schemas/schemas/dp03_1yr.yaml → python/lsst/sdm/schemas/dp03_1yr.yaml b/...on/lsst/sdm_schemas/schemas/dp03_1yr.yaml → python/lsst/sdm/schemas/dp03_1yr.yaml
diff --git a/python/lsst/sdm_schemas/schemas/hsc.yaml → python/lsst/sdm/schemas/hsc.yaml b/python/lsst/sdm_schemas/schemas/hsc.yaml → python/lsst/sdm/schemas/hsc.yaml
diff --git a/python/lsst/sdm_schemas/schemas/imsim.yaml → python/lsst/sdm/schemas/imsim.yaml b/python/lsst/sdm_schemas/schemas/imsim.yaml → python/lsst/sdm/schemas/imsim.yaml
diff --git a/...chemas/ivoa/ObsCore-v1.1-descriptions.csv → ...chemas/ivoa/ObsCore-v1.1-descriptions.csv b/...chemas/ivoa/ObsCore-v1.1-descriptions.csv → ...chemas/ivoa/ObsCore-v1.1-descriptions.csv
diff --git a/...s/schemas/ivoa/ObsCore-v1.1-mandatory.csv → ...m/schemas/ivoa/ObsCore-v1.1-mandatory.csv b/...s/schemas/ivoa/ObsCore-v1.1-mandatory.csv → ...m/schemas/ivoa/ObsCore-v1.1-mandatory.csv
diff --git a/...as/schemas/ivoa/ObsCore-v1.1-optional.csv → ...dm/schemas/ivoa/ObsCore-v1.1-optional.csv b/...as/schemas/ivoa/ObsCore-v1.1-optional.csv → ...dm/schemas/ivoa/ObsCore-v1.1-optional.csv
diff --git a/...n/lsst/sdm_schemas/schemas/ivoa/README.md → python/lsst/sdm/schemas/ivoa/README.md b/...n/lsst/sdm_schemas/schemas/ivoa/README.md → python/lsst/sdm/schemas/ivoa/README.md
diff --git a/...emas/schemas/ivoa/make_obscore_nominal.py → .../sdm/schemas/ivoa/make_obscore_nominal.py b/...emas/schemas/ivoa/make_obscore_nominal.py → .../sdm/schemas/ivoa/make_obscore_nominal.py
diff --git a/...schemas/schemas/ivoa/obscore_nominal.yaml → ...sst/sdm/schemas/ivoa/obscore_nominal.yaml b/...schemas/schemas/ivoa/obscore_nominal.yaml → ...sst/sdm/schemas/ivoa/obscore_nominal.yaml
diff --git a/...n/lsst/sdm_schemas/schemas/obsloctap.yaml → python/lsst/sdm/schemas/obsloctap.yaml b/...n/lsst/sdm_schemas/schemas/obsloctap.yaml → python/lsst/sdm/schemas/obsloctap.yaml
diff --git a/...sdm_schemas/schemas/oga_live_obscore.yaml → ...on/lsst/sdm/schemas/oga_live_obscore.yaml b/...sdm_schemas/schemas/oga_live_obscore.yaml → ...on/lsst/sdm/schemas/oga_live_obscore.yaml
diff --git a/python/lsst/sdm_schemas/__init__.py b/python/lsst/sdm_schemas/__init__.py
diff --git a/tests/test_schema_resource.py b/tests/test_schema_resource.py
@@ -31,7 +31,7 @@ class SchemaResourceTestCase(unittest.TestCase):
 
     def test_read_resource(self) -> None:
         """Test that schema data can be read from a resource."""
-        resource = importlib.resources.files("lsst.sdm_schemas.schemas").joinpath(
+        resource = importlib.resources.files("lsst.sdm.schemas").joinpath(
             "dp02_dc2.yaml"
         )
         raw_data = resource.read_text()

diff --git a/yml b/yml
@@ -1 +1 @@
-./python/lsst/sdm_schemas/schemas/
+./python/lsst/sdm/schemas/
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		./python/lsst/sdm_schemas/schemas/
		./python/lsst/sdm/schemas/