Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement snapshot downloadFiles API call #2797

Merged
merged 11 commits into from
Apr 24, 2023
Merged

Conversation

nellh
Copy link
Contributor

@nellh nellh commented Apr 11, 2023

This creates a new flat API call that is indefinitely cached once a snapshot is fully exported. This is needed to allow for faster downloads without each client repeating tree creation.

Changes:

  • Adds a doNotCache options to the CacheItem miss function to allow for a cache miss returning valid data to still avoid the cache
  • Uses the doNotCache flag to not cache any file trees that are not from S3. This is probably the biggest change here because draft performance may suffer, it might make sense to add a flag for drafts to not give up caching them entirely but prevent their inclusion in the snapshotDownload cache before an export is complete.
  • Adds Snapshot.downloadFiles which follows the existing DatasetFile interface.
  • Limits the cacheClear mutation to the snapshotDownload item type only.
  • Deprecates the unused DatasetChanges model.

@codecov
Copy link

codecov bot commented Apr 11, 2023

Codecov Report

Merging #2797 (ecebe31) into master (7e0baf0) will decrease coverage by 0.06%.
The diff coverage is 67.89%.

@@            Coverage Diff             @@
##           master    #2797      +/-   ##
==========================================
- Coverage   75.83%   75.77%   -0.06%     
==========================================
  Files         411      409       -2     
  Lines       67139    67087      -52     
  Branches     2039     2034       -5     
==========================================
- Hits        50912    50838      -74     
- Misses      16176    16199      +23     
+ Partials       51       50       -1     
Impacted Files Coverage Δ
...es/openneuro-server/src/graphql/resolvers/query.js 100.00% <ø> (ø)
...euro-server/src/graphql/resolvers/subscriptions.js 92.59% <ø> (-0.59%) ⬇️
packages/openneuro-server/src/models/dataset.ts 100.00% <ø> (+12.63%) ⬆️
packages/openneuro-server/src/cache/item.ts 71.27% <18.75%> (-5.47%) ⬇️
packages/openneuro-server/src/datalad/snapshots.js 61.53% <24.24%> (-4.63%) ⬇️
...es/openneuro-server/src/graphql/resolvers/cache.ts 28.57% <38.46%> (+16.07%) ⬆️
...penneuro-server/src/graphql/resolvers/snapshots.js 36.50% <66.66%> (+0.10%) ⬆️
packages/openneuro-server/src/datalad/files.ts 87.71% <87.71%> (ø)
packages/openneuro-server/src/cache/types.ts 100.00% <100.00%> (ø)
...ckages/openneuro-server/src/datalad/description.js 74.32% <100.00%> (ø)
... and 5 more

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@nellh
Copy link
Contributor Author

nellh commented Apr 11, 2023

After some discussion, I think we should instead make the file tree cache stable and put the export URLs in their own cache that is invalidated when the export status changes.

@nellh
Copy link
Contributor Author

nellh commented Apr 19, 2023

After some discussion, I think we should instead make the file tree cache stable and put the export URLs in their own cache that is invalidated when the export status changes.

Going back on this idea after some additional development on it, it leads to a difficult to resolve pattern where the relationship between the git tree objects and the URLs can get stale that isn't easy to resolve without dropping the draft cache anyways. The existing solution works pretty well, if it does cause real performance issues with drafts we could set an expiration time on the unexported draft tree cache but I think this version should be measured first.

@nellh nellh merged commit 4d7f7f1 into master Apr 24, 2023
@nellh nellh deleted the download-api-improvements branch April 24, 2023 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant