Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor clean task #1805

Merged
merged 4 commits into from
Oct 11, 2021
Merged

Conversation

WithoutPants
Copy link
Collaborator

The existing clean task code found all scenes, images and galleries. This caused a massive amount of memory usage. The clean task has been refactored to perform clean assessments in query batches of 1000. This means that (unfortunately) items must be removed after assessing all of the items of that type, otherwise the batch query will be impacted.

@WithoutPants WithoutPants added the bug Something isn't working label Oct 5, 2021
@WithoutPants WithoutPants added this to the Version 0.11.0 milestone Oct 5, 2021
@bnkai
Copy link
Collaborator

bnkai commented Oct 6, 2021

Code looks ok.
It seems to work ok also from a few quick tests.

A further optimization (out of scope of the PR) would be to not use the Query function.
While the Query function is ok for general usage, imho for maintenance or batch tasks that are very specific it may be too generic.
For example a simple SELECT path,id from scenes LIMIT 1000 OFFSET ? query would be a lot faster and lighter in this case.
For clean tasks we only need the id and path from every file. Using the Query function means that we get all data (and with a lot more db requests), which for the scenes (long descriptions) could be a considerable amount of bytes per batch.

@kermieisinthehouse
Copy link
Collaborator

I deleted a folder and ran a clean task on this branch.

I can confirm :

  • it didn't drop my entire database
  • it didn't drop anything that's still there, as far as the logs go
  • the scenes from the folder I deleted are gone

@WithoutPants WithoutPants merged commit 46ae458 into stashapp:develop Oct 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants