Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Does dropPrefix remove !badger!move keys? #1288

Closed
sana-jawad opened this issue Apr 4, 2020 · 4 comments
Closed

Does dropPrefix remove !badger!move keys? #1288

sana-jawad opened this issue Apr 4, 2020 · 4 comments
Labels
area/gc Garbage Collection issues. kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) status/accepted We accept to investigate or work on it.

Comments

@sana-jawad
Copy link
Contributor

What version of Go are you using (go version)?

$ go version
go version go1.13.6

What version of Badger are you using?

2.0.2

Does this issue reproduce with the latest master?

yes

What are the hardware specifications of the machine (RAM, OS, Disk)?

16 GB, Mac OS, 2TB

What did you do?

  1. Inserted 1M of keys in badgerDB with two common prefixes say prefix1, prefix2
  2. Called dropPrefix for prefix1 which resulted in removing 500K keys.
  3. ValueLogGC discard ratio is set to 0.3 and frequency of calling ValueLogGC is 1min.
  4. There were no badger!move keys till this point. When ValueLogGC got triggered, it cleaned up some of the value log files.
  5. After the clean up !badger!move keys showed up in keys histogram.
  6. Called dropPrefix on prefix2 as well.
  7. There are no keys in DB but only !badger!move keys are still there.
  8. So ValueLog files are not cleaned up entirely.

What did you expect to see?

All ValueLog files except one would be cleaned up and there are no internal move keys.

What did you see instead?

!badger!move keys are not cleaned up.
@jarifibrahim
Copy link
Contributor

Hey @sana-jawad, we do not clean up the badger!move! keys. This is a known issue. One of the challenges here is to figure out which !badger!move keys are actually invalid and which ones are valid.

We could perform lookups for all !move! keys but this would severely affect the write performance as the compactions would become very expensive. This is a fairly tough task and we're working on finding a solution for it.

@jarifibrahim jarifibrahim added area/gc Garbage Collection issues. kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) status/accepted We accept to investigate or work on it. labels Apr 6, 2020
@sana-jawad
Copy link
Contributor Author

Thanks @jarifibrahim for the reply. It makes sense. I have a suggestion for dropPrefix method:

When dropPrefix is called for a "prefix", the dropPrefix should also drop keys with !badger!move+prefix. For drop prefix the choice is much more easy to make. Right now as a work around I am calling dropPrefix two times:
dropPrefix(prefix)
dropPrefix(!badger!move+prefix)

Also an observation, lower the value of discardRatio the higher the number of !badger!move keys. And thus more the number of !badger!move keys, the size on disk keeps on increasing over time. It may be added in the comments for discardRatio.

@jarifibrahim
Copy link
Contributor

Thanks @sana-jawad. The dropPrefix(!badger!move+prefix) seems like a decent idea. Feel free to send a PR for it and the documentation changes. :)

@sana-jawad sana-jawad mentioned this issue Apr 17, 2020
Merged
sana-jawad added a commit to salesforce/sloop that referenced this issue Apr 24, 2020
@sana-jawad
Sloop was not able to reduce the size on disk after GC runs. This als…
c595202 
…o plays a big role in memory consumption. (#114)

Upgraded Badger version.
Added flattening at start up time.
Fixed the event count spreading issue which resulted in uneven data distribution across partitions.
Moved to drop prefix as it yields better space claim.
Added feature flag for switching to delete prefix. Also changed the numberfversions to 0 so that delete prefix would reclaim space. hypermodeinc/badger#1228
Fixed the issue of unclaimed !badger!move prefixes which are never cleaned up. Details: hypermodeinc/badger#1288
Added support in debugging pages to see internal keys.
@sana-jawad sana-jawad mentioned this issue May 15, 2020
Merged
@jarifibrahim
Copy link
Contributor

This has been fixed via #1331.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/gc Garbage Collection issues. kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) status/accepted We accept to investigate or work on it.
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jarifibrahim @sana-jawad