Skip to content

Commit

Permalink
Release: v0.2.0 (#424)
Browse files Browse the repository at this point in the history
* fix typo when setting model level

* fix an import bug and clean up unit test_scenario_1

* fix a couple import bugs

* update files with new accurate pressure model delays

* fix a pressure level specification bug in era5

* Update files

* add in ecef2enu

* start some new unit tests for test_util

* comment out extra LOS calcs (From LOS file)

* Add in ecef2enu

* convert ECEF unit vectors to ENU

* Fix LOS calculation and remove print statements

* add a slant range threshold for sanity checking los

* Ensure that the geo2rdr object is deleted

* update unit tests and fix a bug

* Bunch of clean up and making everything consistent

* fix a couple bugs

* Add unit tests and docstrings

* refactor losreader to be cleaner, add unit tests

* clarify a couple points in losreader

* start refactoring LOS calc

* move some files around

* BB_few_fixes

change name of variable (los_file -> lookvector)
fix masking with a reshape
make shape of outputs consistent (LOS file, orbit file)

* Ensure that input shape is preserved in losreader

* fix and write

* remove extra write

* test fixes from Jeremy

* Fix call to getLookVectors

* Fix some bugs and make sure los vars get carried through correctly

* Fix unit tests for GMAO and a few bugs

* autopep8 fixes

* add in dummy file; github does not like using dummy.txt

* update unit test

* use dem_stitcher

* add new line add end of environment.yml

* remove extraneous code

* update env-dev

* put new dependencies into pip

* add geopandas

* remove extra functions and update docstring of getDEM

* remove DEM download from test suite

* Add a mamba v. 2 (#323)

* update dependencies

* fix a typo

* Take care of GNSS duplicates

* Fix how dups are dropped and add a unit test for it

* remove unneeded test breaking

* add fixed zlevels to other weather models ie non ECMWF

* identify tests that are long

* Specify tests as long and skip on circleci

* skip tests that require file download if they do not exist

* skip tests that require file download if they do not exist

* Fix bugs and remove old breakpoint statements

* Pass closest localtime as opposed to mean within range

* fix a few typos

* Fix a few more bugs, and finish off the conventional calculation

* updated WeatherModes.md with revised information

* fix import statements in the unit test suite

* Fix local-time bug

* Remove reference to local env

* fix a bug for parsing los

* Fix a bug when reading lat/lon files

* Fix bug, was expecting localtime

The code had expected a localtime key, which led to crashes in cases where a user does not wish to perform localtime analyses

* remove breakpoint from gmao.py

* update dem.py with new dem-stitcher api

* update getDEM to use the new dem-stitcher API

* Fix dem-stitcher version for the current API

* Add dev dep "autopep8"

The PR template suggests running it on your changes.

* autopep8

* flake8

* flake8

* flake8

* install via conda instead of pip

* Remove unused writeWeatherVars2HDF5()

* flake8

* Install via conda instead of pip

* Raise better error for missing subdatasets

* flake8

Somehow these changes flew under the radar. Yes, I am aware of the irony of that statement.

* move all pip deps except dem-stitcher to conda deps

* Update dev env file to reflect dem-stitcher update

* Update env file to reflect dem-stitcher update

* Update dev env file to newest dem-stitcher

* Update env file to newest dem-stitcher

* stash dep changes

* update environment deps

* correct name of dem_stitcher conda package

* remove uneeded deps

* Update ERA5 API

* autopep8

* Version 0.1.0

* Update readme, clean up old files (#360)

* Initial readme update

* clean up old files,readme

* fix contrib typos

Co-authored-by: Jeremy Maurer <[email protected]>

* Adding env file for using isce3

* Remove use of geometry extension

* Pulling in #332 and updating tests

* Pull Earthdata from netrc

* Using UTC time tags and removing ref time for state vectors

* Use instance to check for zenith - working zenith tests

* Fixing typos. Working LOS tests

* Using isce3 for look angle computation - Conventional

* Fix for unit tests

* Making isce3 the default conda envs

* Controlling import of isce3 to circumvent pycoverage's double import

* Trying CI with isce3 pinned

* Quick bump up of python version

* change to config file format

* few tweaks to cfg file

* autopep8 suggestions

* Adding ECEF look vector computation

* Adding Top of Atmosphere calculation

* Adding simple unit test for raytracing

* updates for new yaml file config

* Refactoring getTopOfAtmosphere

* Refactoring to use bounds only always to load models

* Fix for unit tests for llreader

* Fix bug in managing GDAL config option

* update raider config yaml

* Completely replace gdal with rasterio + pyproj

* Changes based on feedback

* Use rasterio profile everywhere

* PEP8 cleanup

* fix spacing

* update checkArgs to work with yaml input

* Carry through config file changes to dem.py

* Convert to entrypoints instead of scripts for better portability

* intiial switch to pyproject.toml

* drop hashbangs

* clean up setup.py

* tweak circleci

* really, really remove environemnt-dev.yml

* Add tests for entrypoints

* drop redundant entrypoint tests from circle ci

* Cube generation prototype

* Feedback from review

* remove cartopy as its causing an issue and fix a few typos

* fix fwe typos

* Transition scripts to entry points

Added a main to every one of the modules we will be using for an entry point and used said main function for the entry points in the setup.py.

* ZTD for bounding box is workinggit status

* fix typo

* update test suite with new APIs and temporarily disable IO tests

* fix bug in test suite

* remove depracated tests

* Ray tracing cube prototype

* Getting golden tests to work with new parser

* Checking in cube generation

* Removing prepareCube

* Fixes for cubes in non lat/lon systems

* Minor optimization of ray tracing to speed up computation

* Documenting ray intersection approach

* Add dockerfile

* Use git object name for container tag

* Add authors and drop vendor from docker image labels

* add dependabot to keep actions up to date

* Fix hrrr (#379)

* update default yaml

* initial try at updating hrrr

* initial working HRRR

* need to keep projection info for HRRR

* add herbie to environment.yml

* Update environment.yml

Co-authored-by: Jeremy Maurer <[email protected]>

* Update Dockerfile

Co-authored-by: dbekaert <[email protected]>

* remove a few typos

* Fix some typos, add a _time_res attribute to weather models

* Fix some bugs and get basic ztd working

* add geocoded_file and move all cube calcs to cube function

* fix file naming bug that was keeping delay cube from being written

* fix but in test suite

* update unit tests

* fix bug in drop_nans

* fix nan-dropping bug

* fix gmao api call

* add back runtime parameter parsing

* Update build.yml

* Prepping for chunking with multiprocessing

* Making output arrays optional

* Multiprocessing work in progress

* Fixes for handling model info

* Updated HRRR subsetter

* Update packaging to use pyproj and generate a dynamic version number (#385)

* convert most project metadata to pyproject.toml

* add setuptools_scm to environment

* drop version.txt

* Add __version__ to package

* force conda-forge

* tag docker with python version too

* Update pyproject.toml

Co-authored-by: dbekaert <[email protected]>

* Fix typo

Co-authored-by: dbekaert <[email protected]>

* Updating HRRR with subsetting and correct conventions

* Adding support to write netcdf in other projections

* Minor cleanup

* Write support for HRRR

* Working HRRR Zenith Delays

* Working ray tracing for HRRR

* Addressing feedback

* automatically extend buffer when raytracing

* clarify between conventional std and raytracing

* Need to return a list of arrays with multi-band rasters

* add variable to context manager so hgts are retrievied

* updates needed for RAiDER_tutorial notebook (#394)

* add jupyter to conda install

* add wand for pdf viewing in jupyter

* check for heights in actual dataframe

* fix bug in writing delays for gnss station file

* correct output filename for stations

* automatically change file extension to nc if not nc or h5

* force rasterio version to 3.3

* make rasterio >= 1.3.3

* begin support for cli interface to dlGNSS and specific steps for delay (#401)

* begin support for cli interface to dlGNSS and specific steps for delay

* doStep is now supported and enables GNSS download

* doStep now fully works

* download GNSS stations works

* change name of gps output dir

* fix HRRR z-level sign issue by using native levels (#406)

Co-authored-by: Jeremy Maurer <[email protected]>

* support reading the AOI options (#402)

* support reading lat/lons from a GUNW

* enable calculation of delay with a GUNW file and specified height levels

* extra check to make sure GNSS file is treated separately

* update test to reflect change in output of rio_extent (from WSEN to SNWE)

* match type for updated test

* enable reading of bbox from ISCE geocoded lat/lon files

* calc delays at DEM heights

* remove accidentally addited GNSS cli

* Projection bounds (#409)

* validator bug fix

* support for reusing weather model file in projected coords (HRRR)

* remove an if condition and extra comments

* Close to finishing off ztd release (#411)

* make sure the right steps happens and default to only weather model calculation

* Add dem download if hgt file not defined for latlon inputs

* fix a bounding box bug for GMAO

* fix bounding box problem and remove broken models

* Update README.md

Does not need the env to create it the first time.

* Update README.md

Adding Piyush to the README as contributor

* Fix some parsing and extent checking bugs

* Update processWM.py

update log message

* Update runProgram.py

fix for showing debug messages correctly

* Update logger.py

correctly show debug messages when user requests verbose

Co-authored-by: Jeremy Maurer <[email protected]>
Co-authored-by: dbekaert <[email protected]>
Co-authored-by: BB <[email protected]>

* change CLI to raider.py and fns to main (#414)

* change CLI to raider.py and fns to main

* fix fn name in tests

* update circleci; start fixing old functions in downloadGNSS

* replace raiderDelay with raider everywhere

* downloadGNSS notebook and CLI (dostep) works

* relocate downloadGNSS and make it work

* bugfix for reading weather files

* remove steps functionality

* add option for only downloading weather model

* put back files git deleted

* Update README.md

* Update README.md (#415)

* Update README.md

Fix the TOC and hyperlinking.

* describe pulling specific releases (#417)

* Fix CircleCI buid and cleanup config (#422)

* temporary gdal fix

* cleanup circleci config and switch to micromamba

* ensure micromamba is available

* constrain isce3 build

* try revert gdal pin

* Constrain python version for version info workflow

* try * in buildstring

Co-authored-by: Jeremy Maurer <[email protected]>

* Update unit tests (#419)

* update unit tests

* update unit tests and add end-to-end tests

* make end-to-end tests long

* add the orbit file

* take out conventional slant for now

* golden unit tests

* add golden datasets

* update golden datasets paths

* move files

Co-authored-by: Jeremy Maurer <[email protected]>

* Add release automation (#416)

* Add release automation

* Update CHANGELOG.md

fix conventional to not be supported

Co-authored-by: dbekaert <[email protected]>

* remove long test flags (#423)

* remove long test flags

* Skip CDSAPI test for now, passing locally but failing for CircleCI

Co-authored-by: Jeremy Maurer <[email protected]>

Co-authored-by: bbuzz31 <[email protected]>
Co-authored-by: cmarshak <[email protected]>
Co-authored-by: sssangha <[email protected]>
Co-authored-by: Eric Jameson Fielding <[email protected]>
Co-authored-by: Simran S Sangha <[email protected]>
Co-authored-by: Scott Staniewicz <[email protected]>
Co-authored-by: Jeremy Maurer <[email protected]>
Co-authored-by: Jeremy Maurer <[email protected]>
Co-authored-by: the-garlic-os <[email protected]>
Co-authored-by: piyushrpt <[email protected]>
Co-authored-by: Joseph H Kennedy <[email protected]>
Co-authored-by: MLicari1 <[email protected]>
Co-authored-by: dbekaert <[email protected]>
  • Loading branch information
14 people authored Dec 3, 2022
1 parent 3998dad commit 38f14d4
Show file tree
Hide file tree
Showing 189 changed files with 140,319 additions and 15,168 deletions.
63 changes: 63 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
version: 2.1
jobs:
build:
docker:
- image: cimg/base:current
user: root
steps:
- checkout
- run:
name: Setup micromamba
command: |
apt update --yes && apt-get upgrade --yes
apt install -y --no-install-recommends wget ca-certificates git
cd $HOME
curl -Ls https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj bin/micromamba
- run:
name: Setup environment
command: |
eval "$($HOME/bin/micromamba shell hook -s posix)"
micromamba create -f environment.yml
micromamba activate RAiDER
pip install coveralls
echo url: https://cds.climate.copernicus.eu/api/v2 > $HOME/.cdsapirc
echo key: $cdsak >> $HOME/.cdsapirc
echo { > $HOME/.ecmwfapirc
echo ' "url": "https://api.ecmwf.int/v1",' >> $HOME/.ecmwfapirc
echo ' "email": "'$ecmwfu'",' >> $HOME/.ecmwfapirc
echo ' "key": "'$ecmwfk'"' >> $HOME/.ecmwfapirc
echo } >> $HOME/.ecmwfapirc
echo url: $NCUMloc > $HOME/.ncmrlogin
echo username: $NCUMu >> $HOME/.ncmrlogin
echo password: $NCUMp >> $HOME/.ncmrlogin
python --version
python -c "import numpy; print(numpy.__version__)"
python -c "import pyproj; print(pyproj.__version__)"
- run:
name: Install RAiDER and test the install
command: |
eval "$($HOME/bin/micromamba shell hook -s posix)"
micromamba activate RAiDER
python -m pip install .
python -c "import RAiDER; from RAiDER.delay import main"
python -c "import RAiDER; from RAiDER.interpolator import interp_along_axis"
- run:
name: Run unit tests
shell: /bin/bash -l
command: |
eval "$($HOME/bin/micromamba shell hook -s posix)"
micromamba activate RAiDER
COV_OPTIONS=`python -c "import importlib;print(*(' --cov='+p for p in importlib.util.find_spec('RAiDER').submodule_search_locations))"`
pytest -m "not long" test/ $COV_OPTIONS --cov-report=
- run:
name: Report coverage
command: |
eval "$($HOME/bin/micromamba shell hook -s posix)"
micromamba activate RAiDER
python .circleci/fix_coverage_paths.py .coverage $(pwd)/tools/RAiDER/
coverage report -mi
coveralls
23 changes: 23 additions & 0 deletions .circleci/fix_coverage_paths.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import re
import sqlite3
import sys


def main(file, replacement):
conn = sqlite3.connect(file)
cur = conn.cursor()
cur.execute("select path, id from file")

patched = [
(re.sub(r"/.*/RAiDER/", replacement, r[0]), r[1])
for r in cur
]

cur.executemany("update file set path=? where id=?", patched)

conn.commit()
conn.close()


if __name__ == '__main__':
main(*sys.argv[1:])
30 changes: 30 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
name: Bug report
about: Create a report to help us improve
title: "[BUG]"
labels: bug
assignees: ''

---

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Download
2. Command used
3. Error Output

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
- RAiDER git tag: [e.g. git show --summary]
- OS

**Additional context**
Add any other context about the problem her
7 changes: 7 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
# Check for updates to GitHub Actions every weekday
interval: "weekly"
39 changes: 39 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
<!--- Provide a general summary of your changes in the Title above -->

## Description
<!--- Describe your changes in detail -->

## Motivation and Context
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->

## How Has This Been Tested?
<!--- Please run the following command on your PR to ensure code formatting consistency: -->
<!--- autopep8 <changed files/folders> --recursive --in-place --ignore=E5 -->

<!--- Please describe how you tested your changes. -->


<!--- For new functionality, describe added unit tests. -->


<!--- If this PR breaks existing unit tests, please describe in detail -->
<!--- which tests break and why. Describe what functionality is changed. -->


## Screenshots (if appropriate):

## Type of change
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] I have added an explanation of what your changes do and why you'd like us to include them.
- [ ] I have written new tests for your core changes, as applicable.
- [ ] I have successfully ran tests with your changes locally.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
29 changes: 29 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: Docker Build

on:
push:
branches:
- main
- dev
pull_request:
branches:
- main
- dev

jobs:
call-version-info-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]
with:
conda_env_name: RAiDER
python_version: '3.10'

call-docker-ghcr-workflow:
needs: call-version-info-workflow
uses: ASFHyP3/actions/.github/workflows/[email protected]
with:
version_tag: ${{ needs.call-version-info-workflow.outputs.version_tag }}
release_branch: main
develop_branch: dev
user: ${{ github.actor }}
secrets:
USER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
18 changes: 18 additions & 0 deletions .github/workflows/changelog.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: Changelog updated?

on:
pull_request:
types:
- opened
- labeled
- unlabeled
- synchronize
branches:
- main
- dev

jobs:
call-changelog-check-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]
secrets:
USER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
15 changes: 15 additions & 0 deletions .github/workflows/labeled-pr.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: Is PR labeled?

on:
pull_request:
types:
- opened
- labeled
- unlabeled
- synchronize
branches:
- main

jobs:
call-labeled-pr-check-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]
16 changes: 16 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: Create Release

on:
push:
tags:
- 'v*'

jobs:
call-release-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]
with:
release_prefix: RAiDER
develop_branch: dev
sync_pr_label: bot
secrets:
USER_TOKEN: ${{ secrets.RELEASE_ACCESS_TOKEN }}
15 changes: 15 additions & 0 deletions .github/workflows/tag.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: Tag version

on:
push:
branches:
- main

jobs:
call-bump-version-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]
with:
user: dbekaert
email: [email protected]
secrets:
USER_TOKEN: ${{ secrets.RELEASE_ACCESS_TOKEN }}
27 changes: 23 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,7 +1,26 @@
*.swp
__pycache__
.vscode
*.pyc
*~
*.swp
*.DS_Store
.ftpignore
.ftpconfig
.ipynb_checkpoints
build
*.so
*.cmake
CMake*
Makefile
dummy*
weather_files
test/data
*.log
test/ERA*
*.dat
*.pdf
*.pyc
*.cpp
test/*/geom/*.dem
test/*/geom/*.nc
.eggs/
dist/
tools/RAiDER.egg-info/
tools/bindings/utils/makePoints.c
Loading

0 comments on commit 38f14d4

Please sign in to comment.