Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] set a generalized cache path, allowing multiple data caches to be combined downstream #883

Merged

Conversation

zacharyburnett
Copy link
Contributor

@zacharyburnett zacharyburnett commented Jul 16, 2024

since GitHub caching has a quirk where caches are distinguished by both the cache key AND cache path, this should make it much easier to cache WebbPSF data and also CRDS files when running the romancal tests, by setting restore-keys

This PR changes the cache path as so:

- ${{ runner.temp }}/webbpsf-data
+ ${{ runner.temp }}/data/

where webbpsf-data/ is stored within ${{ runner.temp }}/data/

@zacharyburnett zacharyburnett changed the title set a generalized cache path, allowing multiple data caches to be com bined [CI] set a generalized cache path, allowing multiple data caches to be com bined Jul 16, 2024
@zacharyburnett zacharyburnett changed the title [CI] set a generalized cache path, allowing multiple data caches to be com bined [CI] set a generalized cache path, allowing multiple data caches to be combined Jul 16, 2024
@zacharyburnett zacharyburnett force-pushed the ci/general_cache_path branch from 02ee882 to c6ecc04 Compare July 16, 2024 19:41
@zacharyburnett zacharyburnett marked this pull request as ready for review July 16, 2024 19:43
@zacharyburnett zacharyburnett changed the title [CI] set a generalized cache path, allowing multiple data caches to be combined [CI] set a generalized cache path, allowing multiple data caches to be combined downstream Jul 16, 2024
@zacharyburnett zacharyburnett force-pushed the ci/general_cache_path branch from c6ecc04 to 15376a6 Compare July 16, 2024 19:47
@zacharyburnett
Copy link
Contributor Author

@mperrin I apologize for all the notifications; when working on the romancal tests I realized that a more generalized cache path was needed. This and #881 should be the last two polishing PRs and then it should be maintenance-free. Thanks for being so patient!

@mperrin
Copy link
Collaborator

mperrin commented Jul 16, 2024

@zacharyburnett I definitely don't understand the GitHub caching and cache paths at all (I was completely unaware of this functionality before you contributed this PR, to be clear) so it's easy to be patient with lots of PRs. Plus I know how much of a hassle it can be to get working these sorts of infrastructure things which can't be entirely tested locally!

@mperrin mperrin merged commit fb4dd05 into spacetelescope:develop Jul 16, 2024
4 of 8 checks passed
@zacharyburnett zacharyburnett deleted the ci/general_cache_path branch July 17, 2024 12:33
@zacharyburnett
Copy link
Contributor Author

zacharyburnett commented Jul 17, 2024

Thanks! This just needs a manual run of https://github.com/spacetelescope/webbpsf/actions/workflows/download_data.yml to re-make the cache, and then it's all set

EDIT: one more PR to fix the download_data.yml: #885

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants