Skip to content
This repository has been archived by the owner on Aug 2, 2021. It is now read-only.

Garbage collection does not seem to reduce LevelDB datadir folder size #1113

Closed
nonsense opened this issue Jan 14, 2019 · 2 comments
Closed

Garbage collection does not seem to reduce LevelDB datadir folder size #1113

nonsense opened this issue Jan 14, 2019 · 2 comments
Milestone
0.3.12

Comments

@nonsense
Copy link
Contributor

If you start a node with store size of 500, datadir should be about 2MB.

However after multiple uploads and garbage collection runs, I can see that actually datadir gets to 30MB and beyond.

We need to investigate why this is happening and make sure that --store.size param, which is used to limit the disk space used by LevelDB, actually works.
@acud acud self-assigned this Jan 17, 2019
@acud
Copy link
Member

acud commented Feb 15, 2019

So, a bit of research:
I ran a 10 node cluster for 1 day, store size is 1000 chunks (4mb), smoke tests triggered every minute with a large file size.
This was repeated with two clusters, one on edge, the second on a branch with the following leveldb compact call:

@@ -380,7 +381,8 @@ func (s *LDBStore) collectGarbage() error {
        log.Debug("garbage collect done", "c", s.gc.count)

        metrics.GetOrRegisterCounter("ldbstore.collectgarbage.delete", nil).Inc(int64(totalDeleted))
-       return nil
+       return s.db.db.CompactRange(util.Range{})
}

This should essentially call the compact on the entire range of the leveldb store.

This is the output of a monitor loop that checks the datadir size every 30 seconds:
edge:

6.6M	/root/.ethereum/swarm
6.6M	/root/.ethereum/swarm
6.6M	/root/.ethereum/swarm
6.6M	/root/.ethereum/swarm
6.6M	/root/.ethereum/swarm
6.8M	/root/.ethereum/swarm
6.8M	/root/.ethereum/swarm
6.8M	/root/.ethereum/swarm
6.8M	/root/.ethereum/swarm
6.8M	/root/.ethereum/swarm
6.8M	/root/.ethereum/swarm
6.8M	/root/.ethereum/swarm
7.0M	/root/.ethereum/swarm
7.0M	/root/.ethereum/swarm
7.0M	/root/.ethereum/swarm
7.1M	/root/.ethereum/swarm
7.1M	/root/.ethereum/swarm
7.1M	/root/.ethereum/swarm
7.1M	/root/.ethereum/swarm
7.1M	/root/.ethereum/swarm
7.1M	/root/.ethereum/swarm
7.1M	/root/.ethereum/swarm
7.1M	/root/.ethereum/swarm
7.1M	/root/.ethereum/swarm
6.8M	/root/.ethereum/swarm
6.8M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
6.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
7.2M	/root/.ethereum/swarm
7.2M	/root/.ethereum/swarm
7.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
7.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
8.2M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
7.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm
8.3M	/root/.ethereum/swarm

with compaction:

7.6M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
5.9M	/root/.ethereum/swarm
5.9M	/root/.ethereum/swarm
7.5M	/root/.ethereum/swarm
7.5M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
5.8M	/root/.ethereum/swarm
6.0M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.5M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
5.8M	/root/.ethereum/swarm
5.8M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.7M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
7.9M	/root/.ethereum/swarm
5.8M	/root/.ethereum/swarm
5.8M	/root/.ethereum/swarm
7.8M	/root/.ethereum/swarm
8.1M	/root/.ethereum/swarm
8.1M	/root/.ethereum/swarm
8.1M	/root/.ethereum/swarm
8.1M	/root/.ethereum/swarm
8.1M	/root/.ethereum/swarm
8.1M	/root/.ethereum/swarm
5.9M	/root/.ethereum/swarm
5.9M	/root/.ethereum/swarm
6.0M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm
7.7M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
8.0M	/root/.ethereum/swarm
6.0M	/root/.ethereum/swarm
7.6M	/root/.ethereum/swarm

At certain points, major deltas were noticed in the compaction branch data store size (from 8mb to 4mb), however it is not really visible why.

A bug where the number wrong metric was sent to InfluxDB has been submitted to upstream (grafana dashboards were showing no chunks evicted upon GC): ethereum/go-ethereum#19102

In the case that the compaction call is indeed correct - the results are inconclusive so far and I am not sure about the benefits of the actual compaction call upon GCing. I will report more with correct GC metrics merged

@adamschmideg adamschmideg added this to the 0.3.12 milestone Feb 21, 2019
@acud acud removed their assignment Apr 17, 2019
@acud
Copy link
Member

acud commented May 28, 2019

closing this as there's no apparent fix at the moment and localstore size does not leak but saturates at a certain level. let's revisit this topic later on

@acud acud closed this as completed May 28, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.3.12
Development

No branches or pull requests
3 participants
@nonsense @adamschmideg @acud