Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A simpler caching file-system #102

Closed
martindurant opened this issue Aug 9, 2019 · 0 comments
Closed

A simpler caching file-system #102

martindurant opened this issue Aug 9, 2019 · 0 comments

Comments

@martindurant
Copy link
Member

The caching file-system downloads from other file systems to a local store, and uses MMap to only fetch the chunks that are accessed, to a sparse file. An alternative mode of the same class would simply download whole files on first access, potentially with on-the-fly decompression, and then provide the local copy. This could then replace most/all of the functionality in Intake's cache layer (although who takes responsibility for metadata for checkpointing remains to be decided).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant