Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use tox and set up CRDS caching when server is accessible #551

Merged
merged 23 commits into from
Nov 28, 2022
Merged

use tox and set up CRDS caching when server is accessible #551

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
065dfb5
cache CRDS from current operational context
zacharyburnett Jul 27, 2022
51fc6fe
test entire package when using `--pyargs`
zacharyburnett Jul 27, 2022
a9cf4ff
rename jobs
zacharyburnett Jul 27, 2022
5562eef
add entry to change log
zacharyburnett Jul 27, 2022
b2330fa
move CRDS cache to HOME directory
zacharyburnett Jul 27, 2022
f99c656
skip CRDS sync if operational context is blank
zacharyburnett Jul 27, 2022
757938e
add ability to specify CRDS context with `workflow_dispatch`
zacharyburnett Aug 3, 2022
cb3b5ef
use `tox`
zacharyburnett Oct 31, 2022
ae690b8
use `requirements-dev.txt`
zacharyburnett Oct 31, 2022
014ea65
numpy dependency
zacharyburnett Oct 31, 2022
d790d2c
use Roman CRDS ops server when it becomes public
zacharyburnett Oct 31, 2022
bef12ef
save reference files hash for future cache keys
zacharyburnett Oct 31, 2022
58d50cc
only hash files once
zacharyburnett Oct 31, 2022
388d564
add `long_description_content_type` to metadata
zacharyburnett Nov 1, 2022
79fb120
use latest Python in checks and build
zacharyburnett Nov 1, 2022
d447f21
use `pytest-xdist` in bottleneck
zacharyburnett Nov 1, 2022
2e84369
rename jobs
zacharyburnett Nov 1, 2022
9bfee90
remove `pytest-xdist` from older numpy tests
zacharyburnett Nov 1, 2022
fecf768
refactor matrices
zacharyburnett Nov 1, 2022
13022b4
cache `.tox` environment
zacharyburnett Nov 1, 2022
ff16eb9
update change log entry
zacharyburnett Nov 2, 2022
88d6335
don't cache `tox` environment
zacharyburnett Nov 2, 2022
6f7ac0c
use workflow structure from @jdavies-st
zacharyburnett Nov 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
334 changes: 150 additions & 184 deletions .github/workflows/roman_ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,227 +12,193 @@ on:
- main
schedule:
# Weekly Monday 9AM build
# * is a special character in YAML so you have to quote this string
- cron: '0 9 * * 1'
- cron: "0 9 * * 1"

env:
CRDS_SERVER_URL: "https://roman-crds-test.stsci.edu"
CRDS_PATH: "/tmp/crds_cache"
CRDS_SERVER_URL: https://roman-crds.stsci.edu
CRDS_PATH: $HOME/crds_cache
CRDS_CLIENT_RETRY_COUNT: 3
CRDS_CLIENT_RETRY_DELAY_SECONDS: 20

jobs:
style:
name: Code style checks
runs-on: ubuntu-latest
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
id: python
with:
python-version: 3.9
- uses: actions/[email protected]
with:
path: ${{ env.pythonLocation }}
key: style-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml', '**/setup.*') }}
- run: pip install pyproject-flake8
- run: pip freeze
- run: pflake8
audit:
name: Bandit security audit
runs-on: ubuntu-latest
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
id: python
with:
python-version: 3.9
- uses: actions/[email protected]
with:
path: ${{ env.pythonLocation }}
key: audit-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml', '**/setup.*') }}
- run: pip install bandit
- run: pip freeze
- run: bandit romancal -r -x tests,regtest
dependencies:
name: verify dependencies are correct
runs-on: ubuntu-latest
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
id: python
with:
python-version: 3.9
- uses: actions/[email protected]
with:
path: ${{ env.pythonLocation }}
key: dependencies-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml', '**/setup.*') }}
- run: pip install .
- run: pip freeze
- run: verify_install_requires
test:
name: test
needs: [ style, audit, dependencies ]
check:
name: ${{ matrix.name }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ 'ubuntu-latest', 'macos-latest' ]
python: [ '3.8', '3.9', '3.10', '3.11' ]
toxenv: [ check-style, check-security, check-install ]
python-version: [ '3.x' ]
os: [ ubuntu-latest ]
include:
- name: Code style check
toxenv: check-style
- name: Security audit
toxenv: check-security
- name: Verify install_requires in setup.py
toxenv: check-install
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
id: python
with:
python-version: ${{ matrix.python }}
- uses: actions/[email protected]
with:
path: ${{ env.pythonLocation }}
key: test-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml', '**/setup.*') }}
- run: pip install ".[test]" pytest-xdist
- run: pip freeze
- run: pytest -n auto
test_alldeps:
name: test optional dependencies
needs: [ test ]
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- run: tox -e ${{ matrix.toxenv }}
test:
name: ${{ matrix.toxenv }} (Python ${{ matrix.python-version }}, ${{ matrix.os }})
needs: [ check ]
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ 'ubuntu-latest', 'macos-latest' ]
python: [ '3.8', '3.9', '3.10', '3.11' ]
toxenv: [ test-xdist, test-devdeps ]
python-version: [ '3.8', '3.9', '3.10', '3.11' ]
os: [ ubuntu-latest, macos-latest ]
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
id: python
with:
python-version: ${{ matrix.python }}
- uses: actions/[email protected]
with:
path: ${{ env.pythonLocation }}
key: test-alldeps-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml', '**/setup.*') }}
- run: pip install ".[test,all]" pytest-xdist
- run: pip freeze
- run: pytest -n auto
test_devdeps:
name: test developer versions
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- run: >
echo "pmap=$(
curl -s -X POST -d '{"jsonrpc": "1.0", "method": "get_default_context", "params": ["roman"], "id": 1}' ${{ env.CRDS_SERVER_URL }}/json/ |
python -c "import sys, json; print(json.load(sys.stdin)['result'])"
)" >> $GITHUB_OUTPUT
# Get default CRDS_CONTEXT without installing crds client
# See https://hst-crds.stsci.edu/static/users_guide/web_services.html#generic-request
id: crds-context
- uses: actions/cache@v3
with:
path: ${{ env.CRDS_PATH }}
key: crds-${{ steps.crds-context.outputs.pmap }}
- run: tox -e ${{ matrix.toxenv }}
test_older_numpy:
name: ${{ matrix.toxenv }} (Python ${{ matrix.python-version }}, ${{ matrix.os }})
needs: [ test ]
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ 'ubuntu-latest', 'macos-latest' ]
python: [ '3.8', '3.9', '3.10', '3.11' ]
toxenv: [ test-numpy120, test-numpy121, test-numpy120 ]
python-version: [ '3.8', '3.9', '3.10', '3.11' ]
os: [ ubuntu-latest ]
exclude:
- python-version: '3.10'
toxenv: test-numpy120
- python-version: '3.11'
toxenv: test-numpy120
- python-version: '3.11'
toxenv: test-numpy121
- python-version: '3.11'
toxenv: test-numpy122
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
id: python
with:
python-version: ${{ matrix.python }}
- uses: actions/[email protected]
with:
path: ${{ env.pythonLocation }}
key: test-devdeps-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml', '**/setup.*') }}
- run: pip install git+https://github.com/asdf-format/asdf git+https://github.com/spacetelescope/stpipe git+https://github.com/spacetelescope/stdatamodels --pre astropy numpy
- run: pip install ".[test]" pytest-xdist
- run: pip freeze
- run: pytest -n auto
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- run: >
echo "pmap=$(
curl -s -X POST -d '{"jsonrpc": "1.0", "method": "get_default_context", "params": ["roman"], "id": 1}' ${{ env.CRDS_SERVER_URL }}/json/ |
python -c "import sys, json; print(json.load(sys.stdin)['result'])"
)" >> $GITHUB_OUTPUT
# Get default CRDS_CONTEXT without installing crds client
# See https://hst-crds.stsci.edu/static/users_guide/web_services.html#generic-request
id: crds-context
- uses: actions/cache@v3
with:
path: ${{ env.CRDS_PATH }}
key: crds-${{ steps.crds-context.outputs.pmap }}
- run: tox -e ${{ matrix.toxenv }}
test_pyargs:
name: test --pyargs
name: ${{ matrix.toxenv }} (Python ${{ matrix.python-version }}, ${{ matrix.os }}
needs: [ test ]
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ 'ubuntu-latest', 'macos-latest' ]
python: [ '3.8', '3.9', '3.10', '3.11' ]
exclude:
- os: macos-latest
python: '3.11'
toxenv: [ test-pyargs-xdist ]
python-version: [ '3.x' ]
os: [ ubuntu-latest ]
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
id: python
with:
python-version: ${{ matrix.python }}
- uses: actions/[email protected]
with:
path: ${{ env.pythonLocation }}
key: test-pyargs-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml', '**/setup.*') }}
- run: pip install ".[test]" pytest-xdist
- run: pip freeze
- run: pytest -n auto ./docs --pyargs romancal
test_older_numpy:
name: test Numpy ${{ matrix.numpy }} (Python ${{ matrix.python }})
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- run: >
echo "pmap=$(
curl -s -X POST -d '{"jsonrpc": "1.0", "method": "get_default_context", "params": ["roman"], "id": 1}' ${{ env.CRDS_SERVER_URL }}/json/ |
python -c "import sys, json; print(json.load(sys.stdin)['result'])"
)" >> $GITHUB_OUTPUT
# Get default CRDS_CONTEXT without installing crds client
# See https://hst-crds.stsci.edu/static/users_guide/web_services.html#generic-request
id: crds-context
- uses: actions/cache@v3
with:
path: ${{ env.CRDS_PATH }}
key: crds-${{ steps.crds-context.outputs.pmap }}
- run: tox -e ${{ matrix.toxenv }}
test_with_coverage:
name: ${{ matrix.toxenv }} (Python ${{ matrix.python-version }}, ${{ matrix.os }})
needs: [ test ]
runs-on: ${{ matrix.os }}
continue-on-error: true
strategy:
fail-fast: false
matrix:
toxenv: [ test-cov ]
python-version: [ '3.x' ]
os: [ ubuntu-latest ]
python: [ '3.8', '3.9', '3.10', '3.11' ]
numpy: [ '1.20.*', '1.21.*', '1.22.*' ]
exclude:
- python: '3.10'
numpy: '1.20.*'
- python: '3.11'
numpy: '1.20.*'
- python: '3.11'
numpy: '1.21.*'
- python: '3.11'
numpy: '1.22.*'
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
id: python
with:
python-version: ${{ matrix.python }}
- uses: actions/[email protected]
with:
path: ${{ env.pythonLocation }}
key: test-numpy${{ matrix.numpy }}-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml', '**/setup.*') }}
- run: pip install -e ".[test]" pytest-xdist
- run: pip install numpy==${{ matrix.numpy }}
- run: pip freeze
- run: pytest -n auto
test_with_coverage:
name: test with coverage
needs: [ test, test_alldeps, test_devdeps, test_pyargs, test_older_numpy ]
runs-on: ubuntu-latest
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
id: python
with:
python-version: 3.9
- uses: actions/[email protected]
with:
path: ${{ env.pythonLocation }}
key: test-coverage-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml', '**/setup.*') }}
- run: pip install ".[test]" pytest-xdist pytest-cov
- run: pip freeze
- run: pytest -n auto --cov --cov-report xml --cov-report term-missing
- uses: codecov/codecov-action@v3
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- run: >
echo "pmap=$(
curl -s -X POST -d '{"jsonrpc": "1.0", "method": "get_default_context", "params": ["roman"], "id": 1}' ${{ env.CRDS_SERVER_URL }}/json/ |
python -c "import sys, json; print(json.load(sys.stdin)['result'])"
)" >> $GITHUB_OUTPUT
# Get default CRDS_CONTEXT without installing crds client
# See https://hst-crds.stsci.edu/static/users_guide/web_services.html#generic-request
id: crds-context
- uses: actions/cache@v3
with:
path: ${{ env.CRDS_PATH }}
key: crds-${{ steps.crds-context.outputs.pmap }}
- run: tox -e ${{ matrix.toxenv }}
- if: ${{ contains(matrix.toxenv,'-cov') }}
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
flags: unit
fail_ci_if_error: true
build_docs:
name: build HTML docs
needs: [ test, test_alldeps, test_devdeps, test_pyargs, test_older_numpy ]
runs-on: ubuntu-latest
build:
name: ${{ matrix.toxenv }} (Python ${{ matrix.python-version }}, ${{ matrix.os }})
needs: [ test ]
runs-on: ${{ matrix.os }}
strategy:
matrix:
toxenv: [ build-twine ]
python-version: [ '3.8', '3.9', '3.10', '3.11' ]
os: [ ubuntu-latest, macos-latest ]
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
id: python
with:
python-version: 3.9
- uses: actions/[email protected]
with:
path: ${{ env.pythonLocation }}
key: build-docs-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml', '**/setup.*') }}
- run: sudo apt-get install graphviz texlive-latex-extra dvipng
- run: pip install ".[docs]"
- run: pip freeze
- run: sphinx-build -W docs docs/_build
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- run: tox -e ${{ matrix.toxenv }}
4 changes: 3 additions & 1 deletion CHANGES.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,10 @@ photom

- Change photom step to forcibly set the photometric keywords to ``None`` for spectroscopic data. [#591]

tests
-----

- refactor `tox` environment factors and structure GitHub Actions into dependent workflow [#551]

0.8.1 (2022-08-23)
==================
Expand Down Expand Up @@ -97,7 +100,6 @@ tests
- Updated tests to account for the change in dimensionality of the err variable in ramp datamodel. [#520]
- Added SOC tests to check for information available in Level 2 images to correct for pixel geometric distortion. [#549]


0.7.1 (2022-05-19)
==================

Expand Down
Loading