Skip to content

Commit

Permalink
use factory from titiler.xarray (#72)
Browse files Browse the repository at this point in the history
import factory and reader from newly released titiler.xarray package
  • Loading branch information
hrodmn authored Jan 9, 2025
1 parent 1f31282 commit 8e6d222
Show file tree
Hide file tree
Showing 63 changed files with 536 additions and 1,104 deletions.
12 changes: 9 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Changelog

## Unreleased

* Import `titiler.xarray` (from [`titiler repo`](https://github.com/developmentseed/titiler)) ([#72](https://github.com/developmentseed/titiler-xarray/pull/72))
* Rename the package to `titiler.multidim` ([#72](https://github.com/developmentseed/titiler-xarray/pull/72)) **breaking change**
* Drop support for kerchunk reference files ([#72](https://github.com/developmentseed/titiler-xarray/pull/72)) **breaking change**
* Drop support for experimental `multiscale` zarr group zoom level functionality ([#72](https://github.com/developmentseed/titiler-xarray/pull/72)) **breaking change**
* Remove default `WebMercatorQuad` tile matrix set in `/tiles`, `/tilesjson.json`, `/map` and `/WMTSCapabilities.xml` endpoints (with upgrade to `titiler.core>=0.19`) **breaking change**

## v0.2.0

### Improved pyramid support through group parameter
Expand All @@ -17,7 +25,7 @@

## v0.1.1

Support for NetCDF and making consolidated metadata optional. See https://github.com/developmentseed/titiler-xarray/pull/39.
Support for NetCDF and making consolidated metadata optional. See <https://github.com/developmentseed/titiler-xarray/pull/39>.

[Performance results between prod (v0.1.0) and dev (unreleased)](https://github.com/developmentseed/tile-benchmarking/blob/bd1703209bbeab501f312d99fc51fda6bd419bf9/03-e2e/compare-prod-dev.ipynb).

Expand All @@ -26,8 +34,6 @@ Support for NetCDF and making consolidated metadata optional. See https://github
* NetCDF Dataset: pr_day_ACCESS-CM2_historical_r1i1p1f1_gn_1950.nc
* Unconsolidated metadata dataset: prod-giovanni-cache-GPM_3IMERGHH_06_precipitationCal


## v0.1.0 (2023-10-11)

Initial release of the project.

12 changes: 5 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# titiler-xarray
# titiler-multidim

---

Expand All @@ -15,10 +15,10 @@ source .venv/bin/activate

python -m pip install -e . uvicorn
export TEST_ENVIRONMENT=true # set this when running locally to mock redis
uvicorn titiler.xarray.main:app --reload
uvicorn titiler.multidim.main:app --reload
```

To access the docs, visit http://127.0.0.1:8000/api.html.
To access the docs, visit <http://127.0.0.1:8000/api.html>.
![](https://github.com/developmentseed/titiler-xarray/assets/10407788/4368546b-5b60-4cd5-86be-fdd959374b17)

## Testing
Expand All @@ -29,13 +29,13 @@ To run all the tests:

```bash
python -m pip install -e ".[test]"
python -m pytest --cov titiler.xarray --cov-report term-missing -s -vv
python -m pytest
```

To run just one test:

```bash
python -m pytest tests/test_app.py::test_get_info --cov titiler.xarray --cov-report term-missing -s -vv
python -m pytest tests/test_app.py::test_get_info
```

## VEDA Deployment
Expand All @@ -46,7 +46,6 @@ The Github Actions workflow defined in [.github/workflows/ci.yml](./.github/work
* The production stack is deployed when the `main` branch is tagged, creating a new release. The production stack will deploy to a stack with an API Gateway associated with the domain prod-titiler-xarray.delta-backend.com/.
* The development stack will be deployed upon pushes to the `dev` and `main` branches. The development stack will deploy to a stack with an API Gateway associated with the domain dev-titiler-xarray.delta-backend.com/.


## New Deployments

The following steps detail how to to setup and deploy the CDK stack from your local machine.
Expand Down Expand Up @@ -94,7 +93,6 @@ The following steps detail how to to setup and deploy the CDK stack from your lo
AWS_DEFAULT_REGION=us-west-2 AWS_REGION=us-west-2 AWS_PROFILE=smce-veda STACK_STAGE=production npm --prefix infrastructure/aws run cdk -- deploy titiler-xarray-production
```


**Important**

In AWS Lambda environment we need to have specific version of botocore, S3FS, FSPEC and other libraries.
Expand Down
2 changes: 1 addition & 1 deletion infrastructure/aws/lambda/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ RUN yum install -y gcc-c++
COPY pyproject.toml pyproject.toml
COPY LICENSE LICENSE
COPY README.md README.md
COPY titiler/ titiler/
COPY src/titiler/ src/titiler/

# Install dependencies
# HACK: aiobotocore has a tight botocore dependency
Expand Down
2 changes: 1 addition & 1 deletion infrastructure/aws/lambda/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

from mangum import Mangum

from titiler.xarray.main import app
from titiler.multidim.main import app

logging.getLogger("mangum.lifespan").setLevel(logging.ERROR)
logging.getLogger("mangum.http").setLevel(logging.ERROR)
Expand Down
36 changes: 24 additions & 12 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "titiler.xarray"
description = "TiTiler extension for xarray."
name = "titiler-multidim"
description = "TiTiler application extension for titiler.xarray."
readme = "README.md"
requires-python = ">=3.8"
authors = [
Expand All @@ -25,6 +25,8 @@ classifiers = [
]
dynamic = ["version"]
dependencies = [
"titiler.core>=0.19.0,<0.20",
"titiler.xarray>=0.19.0,<0.20",
"cftime",
"h5netcdf",
"numpy<2.0.0",
Expand All @@ -36,25 +38,27 @@ dependencies = [
"s3fs",
"aiohttp",
"requests",
"pydantic==2.0.2",
"titiler.core>=0.14.1,<0.15",
"pydantic-settings~=2.0",
"pydantic>=2.4,<3.0",
"pandas==1.5.3",
"redis",
"fastapi>=0.100.0,<0.107.0",
"starlette<0.28",
"fastapi>=0.108.0,<0.109.0",
"starlette>=0.29.0,<0.33.0",
]

[project.optional-dependencies]
test = [
"pytest",
"pytest-cov",
"pytest-asyncio",
"httpx",
"httpx<0.28",
"yappi",
]
dev = [
"pre-commit"
"dask>=2023.5.0",
"ipython>=8.12.3",
"netcdf4>=1.7.2",
"pre-commit",
]
debug = [
"yappi"
Expand Down Expand Up @@ -111,13 +115,21 @@ namespace_packages = true
explicit_package_bases = true

[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"
requires = ["pdm-backend"]
build-backend = "pdm.backend"


[tool.pdm.version]
source = "file"
path = "titiler/xarray/__init__.py"
path = "src/titiler/multidim/__init__.py"


[tool.pdm]
package-dir = "src/"

[tool.pdm.build]
includes = ["titiler/xarray"]
includes = ["src/titiler/multidim/"]
excludes = ["tests/", "**/.mypy_cache", "**/.DS_Store"]

[tool.pytest.ini_options]
addopts = "--cov=titiler.multidim --cov-report term-missing -s -vv"
3 changes: 3 additions & 0 deletions src/titiler/multidim/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
"""titiler.multidim"""

__version__ = "0.3.0"
181 changes: 181 additions & 0 deletions src/titiler/multidim/factory.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
"""TiTiler.xarray factory."""

from typing import List, Literal, Optional, Type
from urllib.parse import urlencode

import jinja2
import numpy as np
from attrs import define
from fastapi import Depends, Query
from starlette.requests import Request
from starlette.responses import HTMLResponse
from starlette.templating import Jinja2Templates
from typing_extensions import Annotated

from titiler.core.dependencies import ColorFormulaParams, DefaultDependency
from titiler.core.resources.enums import ImageType
from titiler.core.resources.responses import JSONResponse
from titiler.multidim.reader import XarrayReader
from titiler.xarray.dependencies import DatasetParams, XarrayIOParams, XarrayParams
from titiler.xarray.factory import TilerFactory as BaseTilerFactory


@define(kw_only=True)
class XarrayTilerFactory(BaseTilerFactory):
"""Xarray Tiler Factory."""

reader: Type[XarrayReader] = XarrayReader
reader_dependency: Type[DefaultDependency] = XarrayParams
dataset_dependency: Type[DefaultDependency] = DatasetParams

def register_routes(self) -> None: # noqa: C901
"""Register Info / Tiles / TileJSON endoints."""
super().register_routes()
self.variables()

def variables(self) -> None:
"""Register /variables endpoint"""

@self.router.get(
"/variables",
response_class=JSONResponse,
responses={200: {"description": "Return dataset's Variables."}},
)
def get_variables(
src_path=Depends(self.path_dependency),
io_params=Depends(XarrayIOParams),
) -> List[str]:
"""return available variables."""
return self.reader.list_variables(
src_path=src_path,
group=io_params.group,
decode_times=io_params.decode_times,
)

def statistics(self) -> None:
"""Register /statistics and /histogram endpoints"""
super().statistics()

@self.router.get(
"/histogram",
response_class=JSONResponse,
responses={200: {"description": "Return histogram for this data variable"}},
response_model_exclude_none=True,
)
def histogram(
src_path=Depends(self.path_dependency),
reader_params=Depends(self.reader_dependency),
):
with self.reader(
src_path=src_path,
variable=reader_params.variable,
group=reader_params.group,
decode_times=reader_params.decode_times,
datetime=reader_params.datetime,
) as src_dst:
boolean_mask = ~np.isnan(src_dst.input)
data_values = src_dst.input.values[boolean_mask]
counts, values = np.histogram(data_values, bins=10)
counts, values = counts.tolist(), values.tolist()
buckets = list(
zip(values, [values[i + 1] for i in range(len(values) - 1)])
)
hist_dict = []
for idx, bucket in enumerate(buckets):
hist_dict.append({"bucket": bucket, "value": counts[idx]})
return hist_dict

def map_viewer(self) -> None:
"""Register /map endpoints"""

@self.router.get("/{tileMatrixSetId}/map", response_class=HTMLResponse)
def map_viewer(
request: Request,
tileMatrixSetId: Annotated[ # type: ignore
Literal[tuple(self.supported_tms.list())],
"Identifier selecting one of the supported TileMatrixSetIds",
],
url: Annotated[Optional[str], Query(description="Dataset URL")] = None,
variable: Annotated[
Optional[str],
Query(description="Xarray Variable"),
] = None,
group: Annotated[
Optional[int],
Query(
description="Select a specific zarr group from a zarr hierarchy, can be for pyramids or datasets. Can be used to open a dataset in HDF5 files."
),
] = None,
decode_times: Annotated[
bool,
Query(
title="decode_times",
description="Whether to decode times",
),
] = True,
drop_dim: Annotated[
Optional[str],
Query(description="Dimension to drop"),
] = None,
datetime: Annotated[
Optional[str], Query(description="Slice of time to read (if available)")
] = None,
tile_format: Annotated[
Optional[ImageType],
Query(
description="Default will be automatically defined if the output image needs a mask (png) or not (jpeg).",
),
] = None,
tile_scale: Annotated[
int,
Query(
gt=0, lt=4, description="Tile size scale. 1=256x256, 2=512x512..."
),
] = 1,
minzoom: Annotated[
Optional[int],
Query(description="Overwrite default minzoom."),
] = None,
maxzoom: Annotated[
Optional[int],
Query(description="Overwrite default maxzoom."),
] = None,
post_process=Depends(self.process_dependency),
rescale=Depends(self.rescale_dependency),
color_formula=Depends(ColorFormulaParams),
colormap=Depends(self.colormap_dependency),
render_params=Depends(self.render_dependency),
dataset_params=Depends(self.dataset_dependency),
):
"""Return map Viewer."""
jinja2_env = jinja2.Environment(
loader=jinja2.ChoiceLoader([jinja2.PackageLoader(__package__, ".")])
)
templates = Jinja2Templates(env=jinja2_env)

if url:
tilejson_url = self.url_for(
request, "tilejson", tileMatrixSetId=tileMatrixSetId
)
if request.query_params._list:
tilejson_url += f"?{urlencode(request.query_params._list)}"

tms = self.supported_tms.get(tileMatrixSetId)
return templates.TemplateResponse(
name="map.html",
context={
"request": request,
"tilejson_endpoint": tilejson_url,
"tms": tms,
"resolutions": [matrix.cellSize for matrix in tms],
},
media_type="text/html",
)
else:
return templates.TemplateResponse(
name="map-form.html",
context={
"request": request,
},
media_type="text/html",
)
Loading

0 comments on commit 8e6d222

Please sign in to comment.