Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.0.1 #3

Merged
merged 40 commits into from
Jan 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
5c31b6f
fix gitignore
cmarshak Oct 13, 2023
e48f54b
summarize api
cmarshak Oct 13, 2023
6ecb368
include data files
cmarshak Oct 13, 2023
10b8da4
delete file
cmarshak Oct 13, 2023
8fcc413
include tests and templates for PRs
cmarshak Oct 13, 2023
9368279
fix paths
cmarshak Oct 13, 2023
4bfb71f
fix readme
cmarshak Oct 13, 2023
59cdbf8
rename stitcher!
cmarshak Oct 13, 2023
c7260a4
hansen mosaics and merge performance jump
cmarshak Oct 31, 2023
25c1284
update dem-stitcher
cmarshak Nov 30, 2023
912988e
lint and improve docstring
cmarshak Nov 30, 2023
f34b805
add coherence
cmarshak Dec 4, 2023
a0ebf72
move tile empty exception
cmarshak Dec 4, 2023
a70c76d
add static analysis and changelog workflows
cmarshak Dec 4, 2023
58427e6
lint
cmarshak Dec 4, 2023
5121fca
ensure faster reread of s1-coherence and add test for coh
cmarshak Dec 4, 2023
c8b2289
first changelog
cmarshak Dec 4, 2023
8c71f6e
add version catch
cmarshak Dec 4, 2023
f8ba71a
lint
cmarshak Dec 4, 2023
353aef6
trufflehog
cmarshak Dec 4, 2023
6f2cd17
fix readme
cmarshak Dec 4, 2023
689367f
save notebook
cmarshak Dec 4, 2023
b87b76c
remove outdated comment
cmarshak Dec 4, 2023
a63e9d7
ensure copying is done correctly
cmarshak Dec 4, 2023
07cdbe2
rerun notebooks
cmarshak Dec 4, 2023
3b2560b
rerun notebook
cmarshak Dec 4, 2023
b7a3eb1
add cop100m
cmarshak Dec 8, 2023
eb6dc33
add readme
cmarshak Dec 8, 2023
8a2df00
lint
cmarshak Dec 8, 2023
67a45db
add back s1 coh org nb
cmarshak Dec 9, 2023
9eee1a8
remove print statements
cmarshak Dec 18, 2023
71245a2
workflows
cmarshak Jan 9, 2024
479e496
ruff 1
cmarshak Jan 10, 2024
fe2baad
workflow and 120 line length
cmarshak Jan 10, 2024
bc79e8f
init format
cmarshak Jan 10, 2024
2149358
changelog
cmarshak Jan 10, 2024
11f64ae
update to ruff
cmarshak Jan 10, 2024
e81bf76
Merge pull request #1 from OPERA-Cal-Val/ruff
cmarshak Jan 10, 2024
e887b5b
update token
cmarshak Jan 10, 2024
9f83c44
Merge pull request #2 from OPERA-Cal-Val/release-and-distribute-workf…
cmarshak Jan 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: bug
assignees: ''

---

### The bug
<!-- Please include a clear and concise description of what the bug is. Describe
what actually happened *and* what you expected to happen. -->

### To Reproduce
<!-- Please include the steps to reproduce the behavior. E.g.,
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error -->

### Additional context
<!-- Add any other context or screenshots about the bug here. -->
19 changes: 19 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: enhancement
assignees: ''

---
### Background
<!-- Please include a clear and concise description of *the problem* this feature would solve. -->

### Describe the solution you'd like
<!-- A clear and concise description of what you want to happen. -->

### Alternatives
<!-- A clear and concise description of any alternative solutions or features you've considered. -->

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->
11 changes: 11 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<!--
If this is a pull request for a new release, please use the release template:
{{ cookiecutter.public_url }}/compare/main...develop?template=release.md

If this PR does not include changes that should be reflected in CHANGELOG.md,
please indicate so by affixing the `bumpless` label to this PR.

NOTE: Pull requests should only be opened for merges to protected branches (required) and any
changes which you'd like reviewed. Do not open a pull request to update a feature or personal
branch -- simply merge with `git`.
-->
18 changes: 18 additions & 0 deletions .github/workflows/changelog.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: Changelog updated?

on:
pull_request:
types:
- opened
- labeled
- unlabeled
- synchronize
branches:
- main
- dev

jobs:
call-changelog-check-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]
secrets:
USER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
37 changes: 37 additions & 0 deletions .github/workflows/distribute.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Distribute to PyPI

on:
push:
tags:
- v*

jobs:
distribute:
runs-on: ubuntu-latest
outputs:
SDIST_VERSION: ${{ steps.build.outputs.version }}
steps:
- uses: actions/checkout@v2

- uses: actions/setup-python@v1
with:
python-version: 3.11

- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install --upgrade setuptools build "setuptools-scm[toml]"

- name: Build distribuion
id: build
run: |
git fetch origin +refs/tags/*:refs/tags/*
export SDIST_VERSION=$(python -m setuptools_scm)
echo "::set-output name=version::${SDIST_VERSION}"
python -m build

- name: upload to PyPI.org
uses: pypa/gh-action-pypi-publish@release/v1
with:
user: __token__
password: ${{ secrets.PYPI_TOKEN }}
15 changes: 15 additions & 0 deletions .github/workflows/labeled-pr.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: Is PR labeled?

on:
pull_request:
types:
- opened
- labeled
- unlabeled
- synchronize
branches:
- main

jobs:
call-labeled-pr-check-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]
16 changes: 16 additions & 0 deletions .github/workflows/release-github.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: Create Release

on:
push:
tags:
- 'v*'

jobs:
call-release-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]
with:
release_prefix: tile-stitcher
develop_branch: dev
sync_pr_label: team-bot
secrets:
USER_TOKEN: ${{ secrets.OPERA_PST_GITHUB_TOKEN }}
15 changes: 15 additions & 0 deletions .github/workflows/static-analysis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: Static analysis

on: push

jobs:
call-flake8-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]
with:
local_package_names: tile_stitcher

call-secrets-analysis-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]

call-ruff-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]
35 changes: 35 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Test and Tag

on:
push:
branches:
- main
- dev
pull_request:
branches:
- main
- dev

jobs:
pytest:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11"]

steps:
- uses: actions/checkout@v2

- uses: mamba-org/provision-with-micromamba@main
with:
environment-name: tile-stitcher
environment-file: environment.yml
extra-specs: |
python=${{ matrix.python-version }}


- name: Pytest in conda environment
shell: bash -l {0}
run: |
python -m pip install .
pytest . tests
60 changes: 16 additions & 44 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,6 @@ notebooks_plots/
# Test output
tests/out/

# Data Files
*.zip
*.tar.gz
*.dat
# Include the tile vector data
!dem_stitcher/data/*.geojson.zip

# .pyenv files
.python-version

Expand Down Expand Up @@ -58,30 +51,27 @@ __pycache__/
.envrc

# Gdal and other image formats
notebooks/*.tif
*.tif
*.xml
notebooks/*.shp
notebooks/*.sbx
notebooks/*.sbn
notebooks/*.shx
notebooks/*.prj
notebooks/*.qpj
notebooks/*.dbf
notebooks/*.cpg
notebooks/*.msk
notebooks/*.kmz
notebooks/*.jpg
notebooks/*.wld
notebooks/*.geojson
notebooks/**/*.tif
*.shp
*.sbx
*.sbn
*.shx
*.prj
*.qpj
*.dbf
*.cpg
*.msk
*.kmz
*.jpg
*.wld
*.geojson
*.tif

# VRT
*.vrt
*.wgs84

# Allow tif datacube to be added
!dem_stitcher/data/geoid_18.tif

# Log files
*.log

Expand All @@ -92,22 +82,4 @@ notebooks/**/*.tif
*.pdf

# Ignore json
*.json

# Ignore Experimental Directories
notebooks/experimental/
tests/notebooks/

# Notebooks
*.ipynb
!notebooks/Basic_Demo.ipynb
!notebooks/Comparing_DEMs.ipynb
!notebooks/Staging_a_DEM_for_ISCE2.ipynb
!notebooks/organize_tile_data/0_Format_and_Organize_Data.ipynb
!notebooks/organize_tile_data/1_Missing_glo-30_tiles.ipynb
!notebooks/analysis_and_comparison/0_Alignment_with_original_DEM_tiles.ipynb
!notebooks/analysis_and_comparison/1_Comparison_with_ISCE2_dem.ipynb
!notebooks/Filling_in_missing_glo-30_tiles_with_glo-90_tiles.ipynb
!notebooks/Merging_DEM_Tiles_into_a_VRT.ipynb

notebooks/analysis_and_comparison/la_test/
*.json
3 changes: 3 additions & 0 deletions .trufflehog.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
CHANGELOG.md
.*.ipynb$
.*README.md
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.0.1]
* First release of `tile_stitcher` with following datasets (See readme for links):
- ESA 10m worldcover 2020 and 2021;
- Pekel 30m water occurence 2021;
- S1 coherence from 2020;
- Hansen annual mosaics 2000, 2013 - present
- Cop 100m Landcover 2015-2019
* Includes workflows for static analysis, release to PyPI, and integration testing.
51 changes: 37 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,28 @@ This tool provides a tool to create continuous rasters of global publicly availa

The API can be summarized as
```
from tile_stitcher import get_raster_from_tiles

bounds = [-120.55, 34.85, -120.25, 35.15]
X, p = get_raster_from_tiles(bounds, tile_shortname='esa_world_cover_2021')

# X is an c x m x n numpy array, where c is the number of channels specified by `count`
# p is a dictionary (or a rasterio profile) including relevant GIS metadata; CRS is epsg:4326
```

The rasters are returned in the global lat/lon projection `epsg:4326` and the API assumes that bounds are supplied in this format.

```
import rasterio

with rasterio.open('esa_worlf_cover_2021_subset.tif', 'w', **p) as ds:
ds.write(X)
```

# Installation

In order to easily manage dependencies, we recommend using dedicated project environments
via [Anaconda/Miniconda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html).
via [Anaconda/Miniconda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)
or [Python virtual environments](https://docs.python.org/3/tutorial/venv.html).

1. Clone the repository and navigate to it in the ternmal.
Expand All @@ -20,10 +35,10 @@ or [Python virtual environments](https://docs.python.org/3/tutorial/venv.html).
4. Install the library with `pip`

```
python -m pip install dem_stitcher
python -m pip install .
```

Python 3.10+ is supported.
You can also install for development with `python -m pip install -e .`. Python 3.10+ is supported.

# Notebooks

Expand All @@ -36,22 +51,30 @@ We have notebooks to demonstrate common usage:

The datasets supported are:

1. Pekel
2. ESA World Cover (10 m) 2020 and 2021
```
In [1]: from tile_stitcher.stitcher import DATASET_SHORTNAMES

In [2]: DATASET_SHORTNAMES
Out[2]: ['peckel_water_occ_2021',
'esa_world_cover_2020',
'esa_world_cover_2021',
'hansen_annual_mosaic',
's1_coherence_2020',
'cop_100_lulc_discrete']
```
These correspond to
+ Pekel: https://global-surface-water.appspot.com/download
+ ESA World Cover (10 m) for 2020 and 2021: https://aws.amazon.com/marketplace/pp/prodview-7oorylcamixxc
+ Hansen annual mosaic: https://data.globalforestwatch.org/documents/941f17325a494ed78c4817f9bb20f33a/explore
+ S1 Coherence from December 2019 - Nov 2020: https://aws.amazon.com/marketplace/pp/prodview-iz6lnjbdlgcwa#resources
+ The copernicus 100 m LULC dataset from 2015 - 2019: https://land.copernicus.eu/global/content/annual-100m-global-land-cover-maps-available

See these [notebooks](notebooks/tile_creation) to see how these tiles are generated and organized.

# Dateline support

None curently.

# For Development

This is almost identical to normal installation:

1. Clone this repo `git clone https://github.com/ACCESS-Cloud-Based-InSAR/dem-stitcher.git`
2. Navigate with your terminal to the repo.
3. Create a new environment and install requirements using `conda env update --file environment.yml` (or use [`mamba`](https://github.com/mamba-org/mamba) to speed the install up)
4. Install the package from cloned repo using `python -m pip install -e .`

# Contributing

We welcome contributions to this open-source package. To do so:
Expand Down
Loading
Loading