Add MDIO Schemas and Update Documentation

.pre-commit-config.yaml

@@ -1,12 +1,15 @@
 repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.1.13
+    hooks:
+      - id: ruff-format
+        types_or: [python, pyi]
+      - id: ruff
+        types_or: [python, pyi]
+        args: [--fix]
+
   - repo: local
     hooks:
-      - id: black
-        name: black
-        entry: black
-        language: system
-        types: [python]
-        require_serial: true
       - id: check-added-large-files
         name: Check for added large files
         entry: check-added-large-files
@@ -34,27 +37,6 @@ repos:
         language: system
         types: [text]
         stages: [commit, push, manual]
-      - id: flake8
-        name: flake8
-        entry: flake8
-        language: system
-        types: [python]
-        require_serial: true
-        args: [--darglint-ignore-regex, .*]
-      - id: isort
-        name: isort
-        entry: isort
-        require_serial: true
-        language: system
-        types_or: [cython, pyi, python]
-        args: ["--filter-files"]
-      - id: pyupgrade
-        name: pyupgrade
-        description: Automatically upgrade syntax for newer versions.
-        entry: pyupgrade
-        language: system
-        types: [python]
-        args: [--py38-plus]
       - id: trailing-whitespace
         name: Trim Trailing Whitespace
         entry: trailing-whitespace-fixer

CONTRIBUTING.md

@@ -49,7 +49,7 @@ different systems.
 
 This should seamlessly enable development for users of [VS Code] on systems with docker installed.
 
-### Known Issues:
+### Known Issues
 
 - `git config --global --add safe.directory $(pwd)` might be needed inside the container.
 

docs/reference.md → docs/api_reference.md

@@ -1,4 +1,4 @@
-# Reference
+# API Reference
 
 ## Readers / Writers
 

docs/usage.md → docs/cli_usage.md

@@ -1,4 +1,4 @@
-# Usage
+# Command-Line Usage
 
 ## Ingestion and Export
 
@@ -9,19 +9,19 @@ There are many more options, please see the [CLI Reference](#cli-reference).
 
 ```shell
 $ mdio segy import \
-    -i path_to_segy_file.segy \
-    -o path_to_mdio_file.mdio \
-    -loc 181,185 \
-    -names inline,crossline
+  -i path_to_segy_file.segy \
+  -o path_to_mdio_file.mdio \
+  -loc 181,185 \
+  -names inline,crossline
 ```
 
 To export the same file back to SEG-Y format, the following command
 should be executed.
 
 ```shell
 $ mdio segy export \
-    -i path_to_mdio_file.mdio \
-    -o path_to_segy_file.segy
+  -i path_to_mdio_file.mdio \
+  -o path_to_segy_file.segy
 ```
 
 ## Cloud Connection Strings
@@ -78,7 +78,7 @@ checks them. If it is not pre-authenticated, you need to pass `--storage-options
 Using UNIX:
 
 ```shell
-mdio segy import \
+$ mdio segy import \
   --input-segy-path path/to/my.segy
   --output-mdio-file s3://bucket/prefix/my.mdio
   --header-locations 189,193
@@ -87,8 +87,8 @@ mdio segy import \
 
 Using Windows (note the extra escape characters `\`):
 
-```console
-mdio segy import \
+```shell
+$ mdio segy import \
   --input-segy-path path/to/my.segy
   --output-mdio-file s3://bucket/prefix/my.mdio
   --header-locations 189,193
@@ -113,7 +113,7 @@ authentication information to APIs.
 Using a service account:
 
 ```shell
-mdio segy import \
+$ mdio segy import \
   --input-segy-path path/to/my.segy
   --output-mdio-file gs://bucket/prefix/my.mdio
   --header-locations 189,193
@@ -123,7 +123,7 @@ mdio segy import \
 Using browser to populate authentication:
 
 ```shell
-mdio segy import \
+$ mdio segy import \
   --input-segy-path path/to/my.segy
   --output-mdio-file gs://bucket/prefix/my.mdio
   --header-locations 189,193
@@ -144,7 +144,7 @@ If ADL is not pre-authenticated, you need to pass `--storage-options`.
 `account_key`: Azure Data Lake storage account access key
 
 ```shell
-mdio segy import \
+$ mdio segy import \
   --input-segy-path path/to/my.segy
   --output-mdio-file az://bucket/prefix/my.mdio
   --header-locations 189,193
@@ -196,6 +196,6 @@ get information about usage.
 
 ```{eval-rst}
 .. click:: mdio.__main__:main
-    :prog: mdio
-    :nested: full
+   :prog: mdio
+   :nested: full
 ```

docs/conf.py

@@ -1,30 +1,76 @@
 """Sphinx configuration."""
+
+# -- Project information -----------------------------------------------------
+
 project = "MDIO"
 author = "TGS"
-copyright = "2023, TGS"
+copyright = "2023, TGS"  # noqa: A001
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+
 extensions = [
     "sphinx.ext.autodoc",
     "sphinx.ext.napoleon",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.autosummary",
+    "sphinxcontrib.autodoc_pydantic",
     "sphinx.ext.autosectionlabel",
     "sphinx_click",
     "sphinx_copybutton",
     "myst_nb",
+    "sphinx_design",
 ]
 
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = [
+    "_build",
+    "Thumbs.db",
+    "jupyter_execute",
+    ".DS_Store",
+    "**.ipynb_checkpoints",
+]
+
+intersphinx_mapping = {
+    "python": ("https://docs.python.org/3", None),
+    "numpy": ("https://numpy.org/doc/stable/", None),
+    "pydantic": ("https://docs.pydantic.dev/latest/", None),
+    "zarr": ("https://zarr.readthedocs.io/en/stable/", None),
+}
+
+pygments_style = "vs"
+pygments_dark_style = "material"
+
 autodoc_typehints = "description"
 autodoc_typehints_format = "short"
 autodoc_member_order = "groupwise"
-autoclass_content = "both"
+autoclass_content = "class"
 autosectionlabel_prefix_document = True
 
+autodoc_pydantic_field_list_validators = False
+autodoc_pydantic_field_swap_name_and_alias = True
+autodoc_pydantic_field_show_alias = False
+autodoc_pydantic_model_show_config_summary = False
+autodoc_pydantic_model_show_validator_summary = False
+autodoc_pydantic_model_show_validator_members = False
+autodoc_pydantic_model_show_field_summary = False
+
 html_theme = "furo"
 
 myst_number_code_blocks = ["python"]
 myst_heading_anchors = 2
+myst_words_per_minute = 80
 myst_enable_extensions = [
+    "colon_fence",
     "linkify",
     "replacements",
     "smartquotes",
+    "attrs_inline",
 ]
 
 # sphinx-copybutton configurations

docs/data_models/chunk_grids.md

@@ -0,0 +1,154 @@
+```{eval-rst}
+:tocdepth: 3
+```
+
+```{currentModule} mdio.schemas.chunk_grid
+
+```
+
+# Chunk Grid Models
+
+```{article-info}
+:author: Altay Sansal
+:date: "{sub-ref}`today`"
+:read-time: "{sub-ref}`wordcount-minutes` min read"
+:class-container: sd-p-0 sd-outline-muted sd-rounded-3 sd-font-weight-light
+```
+
+The variables in MDIO data model can represent different types of chunk grids.
+These grids are essential for managing multi-dimensional data arrays efficiently.
+In this breakdown, we will explore four distinct data models within the MDIO schema,
+each serving a specific purpose in data handling and organization.
+
+MDIO implements data models following the guidelines of the Zarr v3 spec and ZEPs:
+
+- [Zarr core specification (version 3)](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html)
+- [ZEP 1 — Zarr specification version 3](https://zarr.dev/zeps/accepted/ZEP0001.html)
+- [ZEP 3 — Variable chunking](https://zarr.dev/zeps/draft/ZEP0003.html)
+
+## Regular Grid
+
+The regular grid models are designed to represent a rectangular and regularly
+paced chunk grid.
+
+```{eval-rst}
+.. autosummary::
+   RegularChunkGrid
+   RegularChunkShape
+```
+
+For 1D array with `size = 31`{l=python}, we can divide it into 5 equally sized
+chunks. Note that the last chunk will be truncated to match the size of the array.
+
+`{ "name": "regular", "configuration": { "chunkShape": [7] } }`{l=json}
+
+Using the above schema resulting array chunks will look like this:
+
+```bash
+ ←─ 7 ─→ ←─ 7 ─→ ←─ 7 ─→ ←─ 7 ─→  ↔ 3
+┌───────┬───────┬───────┬───────┬───┐
+└───────┴───────┴───────┴───────┴───┘
+```
+
+For 2D array with shape `rows, cols = (7, 17)`{l=python}, we can divide it into 9
+equally sized chunks.
+
+`{ "name": "regular", "configuration": { "chunkShape": [3, 7] } }`{l=json}
+
+Using the above schema, the resulting 2D array chunks will look like below.
+Note that the rows and columns are conceptual and visually not to scale.
+
+```bash
+ ←─ 7 ─→ ←─ 7 ─→  ↔ 3
+┌───────┬───────┬───┐
+│       ╎       ╎   │  ↑
+│       ╎       ╎   │  3
+│       ╎       ╎   │  ↓
+├╶╶╶╶╶╶╶┼╶╶╶╶╶╶╶┼╶╶╶┤
+│       ╎       ╎   │  ↑
+│       ╎       ╎   │  3
+│       ╎       ╎   │  ↓
+├╶╶╶╶╶╶╶┼╶╶╶╶╶╶╶┼╶╶╶┤
+│       ╎       ╎   │  ↕ 1
+└───────┴───────┴───┘
+```
+
+## Rectilinear Grid
+
+The [RectilinearChunkGrid](RectilinearChunkGrid) model extends
+the concept of chunk grids to accommodate rectangular and irregularly spaced chunks.
+This model is useful in data structures where non-uniform chunk sizes are necessary.
+[RectilinearChunkShape](RectilinearChunkShape) specifies the chunk sizes for each
+dimension as a list allowing for irregular intervals.
+
+```{eval-rst}
+.. autosummary::
+   RectilinearChunkGrid
+   RectilinearChunkShape
+```
+
+:::{note}
+It's important to ensure that the sum of the irregular spacings specified
+in the `chunkShape` matches the size of the respective array dimension.
+:::
+
+For 1D array with `size = 39`{l=python}, we can divide it into 5 irregular sized
+chunks.
+
+`{ "name": "rectilinear", "configuration": { "chunkShape": [[10, 7, 5, 7, 10]] } }`{l=json}
+
+Using the above schema resulting array chunks will look like this:
+
+```bash
+ ←── 10 ──→ ←─ 7 ─→ ← 5 → ←─ 7 ─→ ←── 10 ──→
+┌──────────┬───────┬─────┬───────┬──────────┐
+└──────────┴───────┴─────┴───────┴──────────┘
+```
+
+For 2D array with shape `rows, cols = (7, 25)`{l=python}, we can divide it into 12
+rectilinear (rectangular bur irregular) chunks. Note that the rows and columns are
+conceptual and visually not to scale.
+
+`{ "name": "rectilinear", "configuration": { "chunkShape": [[3, 1, 3], [10, 5, 7, 3]] } }`{l=json}
+
+```bash
+ ←── 10 ──→ ← 5 → ←─ 7 ─→  ↔ 3
+┌──────────┬─────┬───────┬───┐
+│          ╎     ╎       ╎   │  ↑
+│          ╎     ╎       ╎   │  3
+│          ╎     ╎       ╎   │  ↓
+├╶╶╶╶╶╶╶╶╶╶┼╶╶╶╶╶┼╶╶╶╶╶╶╶┼╶╶╶┤
+│          ╎     ╎       ╎   │  ↕ 1
+├╶╶╶╶╶╶╶╶╶╶┼╶╶╶╶╶┼╶╶╶╶╶╶╶┼╶╶╶┤
+│          ╎     ╎       ╎   │  ↑
+│          ╎     ╎       ╎   │  3
+│          ╎     ╎       ╎   │  ↓
+└──────────┴─────┴───────┴───┘
+```
+
+## Model Reference
+
+:::{dropdown} RegularChunkGrid
+:animate: fade-in-slide-down
+
+```{eval-rst}
+.. autopydantic_model:: RegularChunkGrid
+
+----------
+
+.. autopydantic_model:: RegularChunkShape
+```
+
+:::
+:::{dropdown} RectilinearChunkGrid
+:animate: fade-in-slide-down
+
+```{eval-rst}
+.. autopydantic_model:: RectilinearChunkGrid
+
+----------
+
+.. autopydantic_model:: RectilinearChunkShape
+```
+
+:::