Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache more things in the DB so we don't hit the disk #1083

Closed
f-f opened this issue Oct 14, 2023 · 5 comments · Fixed by #1107
Closed

Cache more things in the DB so we don't hit the disk #1083

f-f opened this issue Oct 14, 2023 · 5 comments · Fixed by #1107
Assignees

Comments

@f-f
Copy link
Member

f-f commented Oct 14, 2023

Now that we have a database, we should not be reading and writing caches on disk, but just add a database table to have them into

@f-f f-f added this to the spago-next alpha bugs milestone Oct 14, 2023
@f-f f-f self-assigned this Oct 16, 2023
@f-f
Copy link
Member Author

f-f commented Oct 16, 2023

We should also cache the results of getManifestFromIndex since those are immutable. (getMetadata is not, since things can get unpublished)

@JordanMartinez
Copy link
Contributor

@finnhodgkin If you're interested, this may be another one worth looking into.

@f-f
Copy link
Member Author

f-f commented Oct 19, 2023

The part where we cache the result of getManifestFromIndex in the database is still very much needed - basically we need a manifest table that whose primary key is a composite of Package name + Version. We'd leave the call sites untouched, and try to look things up in the DB when we need them, going to the disk only when we don't find them in cache

Since we are going to have lockfiles for all builds (see #917), I don't think the part where we cache package sets in the database makes sense anymore

@f-f f-f changed the title Cache package sets in the database rather than on disk Cache more things in the DB so we don't hit the disk Oct 19, 2023
@f-f f-f removed their assignment Oct 19, 2023
@finnhodgkin
Copy link
Contributor

@finnhodgkin If you're interested, this may be another one worth looking into.

I'm not sure how much free time I'll have in the next week or two but happy to take a look if I find some.

@f-f
Copy link
Member Author

f-f commented Oct 23, 2023

Something else to pile on this "bring things in the database": right now we write a file and check its modification time to decide if we should be fetching the registry repo. This is slow, and not portable between OSs, so we should just have a table for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants