Support pebble

[lidel]

TODO

• include https://github.com/ipfs/go-ds-pebble as built-in plugin (similar to badger atm)
• update https://github.com/ipfs/kubo/blob/master/docs/config.md#profiles to include pebble
  • (TBD, sounds like) do we want to allow flatfs (for blocks)+pebble for pins? if so, we need separate profile for applying it only as leveldb replacement, and separate one for /blocks (blockstore)
• update https://github.com/ipfs/kubo/blob/master/docs/datastores.md to include pebble
• (TBD, probably a nice to have but not required) add support to https://github.com/ipfs/ipfs-ds-convert to allow conversion of existing repos?
  • we've decided to not write conversion tools, in practice to switch datastore type, people created new cluster node and transferred data to it, as it was safer and also performed GC of unpinned data for free.
• find a way to deal with ratcheting updates failing (Support pebble  #10347 (comment))
  • expose FormatMajorVersion and write docs in docs/datastores.md#pebbleds ?
• add warning to users running badger, nudging them to switch to flatfs or pebble
• replace leveldb with pebble in the default settings
• soft badger deprecation
  • remove badgerv1 profile, so new users won't be able to use it, unless they had preexisting config
  • detect when badgerv1 is used and refuse to start unless KUBO_ALLOW_DEPRECATED_BADGERV1=true is set in env – creates friction, forcing people to notice

Description

Summary

Include pebble as built-in plugin.
It provides meaningful alternative to leveldb as the datastore, and may be better than badger1 as well.

Background

It is 2024, and we still only have flatfs, leveldb and necromancy-level badgerv1 (!) as datastore options.

We got positive feedback about pebble, some examples:

https://ipfscluster.io/documentation/guides/datastore/

Pebble is a high performant
backend from Cochroachdb, used by default in Cluster:

• Proven to work well on very large pinsets.
• Best disk-usage compared to the rest. No need to trigger GC cycles for space reclaim.
• Performance and memory usage seems on par with Badger3, and behaves better than Badger on both counts.
• Behaves correctly with default settings but we bump them up a bit.
• 0-delay startup times, even with very large amounts of data.
• Options support compression (we chose to leave it enabled by default).
• The Pebble project is officially alive and maintained.
• Pebble only runs on 64-bit architectures.
• One key difference with Badger3 is that Pebble stores keys and values
  together and any lookup for a key will also read the values, while Badger3
  can store keys and values separately (i.e. keys only in the index, which can
  be loaded onto memory when small enough).

ipfs/go-ds-pebble#29:

After changing leveldb store to pebble store, the speed of GC has increased by at least dozens of times. So it's not flatfs that's to blame for slow GC, it's leveldb. I also tried leveldb and pebble as blockstore, but the CPU and memory usage is unacceptable.

Right now, to use go-ds-pebble one needs to build external plugin.

Proposed change

Include https://github.com/ipfs/go-ds-pebble in standard kubo build, just like we do with legacy badger1.

This will

• provide meaningful alternative to leveldb and badgerd
• allow community to use it without having to build external plugin and report back

[guojidan]

hi, I want try implement this feature 😄

[lidel]

Triage note:

• given https://ipfscluster.io/documentation/guides/datastore/ is using pebble by default these days, we are considering bringing Pebble as first-class citizen, similar to legacy badger, and remove badger over time.

[lidel]

Triage notes:

• we want to sunset badgerv1, will start warning people when they apply profile it is deprecated
• we want to enable pebble before removing badgerv1
• we need to figure out which pebble settings are worthy exposing via config, prior art in ipfs-cluster will likely be a good starting point
• we want to track release verion of pebble, to avoid the state we ended up with badgerv1

[hsanjuan]

• we need to figure out which pebble settings are worthy exposing via config, prior art in ipfs-cluster will likely be a good starting point

Expose as much as possible because chances are defaults are never right for everyone.

[gammazero]

@hsanjuan Here is the subset of pebble options that I decided to make configurable.

This decision is based on what CockroachDB configures as pebble defaults, and based on what pebble tuning parameters were useful for optimizing IPNI's use of pebble.

[gammazero]

@lidel I want to stop using ipfs-ds-convert. There should instead be a way to export all data from kubo instance and import it into another. This would free us from ever having to write a conversion for another datastore.

Maybe it could work something like this:

oldipfs export --repo-dir=~/.ipfs_prev --json --stdout | ipfs import --json

[lidel]

@gammazero replacing ipfs-ds-convert with ipfs repo export|import sounds very sensible, will be useful for people running custom datastores as well.

FYSA filled ipfs-inactive/ipfs-ds-convert#50 to track sunsetting ipfs-ds-convert (i'll do it shortly, taking it off our plate)

[hsanjuan]

@hsanjuan Here is the subset of pebble options that I decided to make configurable.

This decision is based on what CockroachDB configures as pebble defaults, and based on what pebble tuning parameters were useful for optimizing IPNI's use of pebble.

I had a quick look. Pebble defaults are quite conservative but should be ok for home use. I had issues with FormatMajorVersion errors during pebble upgrades: https://github.com/ipfs-cluster/ipfs-cluster/pull/2019/files . When not explicit, the current default is used. Sometimes ratcheting the database after an upgrade fails if you jumped too many pebble releases in-between.

[lidel]

Added FormatMajorVersion to the TODO at the top of the issue + reopening to track next steps.

[hsanjuan]

So, I put a collab cluster peer on Pebble. It should pin 3.3T but after 72h it has used 1.1T, and only 300GiB seem to be useful data according to Pebble stats:

|                             |       |       |   ingested   |     moved    |    written   |       |    amp   |     multilevel
 level | tables  size val-bl vtables | score |   in  | tables  size | tables  size | tables  size |  read |   r   w  |    top   in  read
 ------+-----------------------------+-------+-------+--------------+--------------+--------------+-------+----------+------------------
     0 |    49   43MB     0B       0 |  0.50 | 316GB |     0     0B |     0     0B |  366K  312GB |    0B |  12  1.0 |    0B    0B    0B
     1 |     0     0B     0B       0 |  0.00 |    0B |     0     0B |     0     0B |     0     0B |    0B |   0  0.0 |    0B    0B    0B
     2 |    75  260MB     0B       0 |  1.68 | 243GB |     0     0B |     0     0B |  435K  1.5TB | 1.5TB |   1  6.4 |    0B    0B    0B
     3 |   334  1.4GB     0B       0 |  1.45 | 292GB |     0     0B |   923  2.1GB |  227K  983GB | 983GB |   1  3.4 |    0B    0B    0B
     4 |   959  8.1GB     0B       0 |  1.46 | 289GB |     0     0B |  3.0K  4.4GB |  116K  849GB | 849GB |   1  2.9 |  11GB  36GB 105GB
     5 |  2.9K   46GB     0B       0 |  1.46 | 280GB |     0     0B |  2.1K  4.5GB |   61K  824GB | 824GB |   1  2.9 |  16GB  54GB 162GB
     6 |  6.3K  256GB     0B       0 |     - | 256GB |     0     0B |    71   22MB |   24K  754GB | 754GB |   1  2.9 |  17GB  62GB 181GB
 total |   11K  312GB     0B       0 |     - | 316GB |     0     0B |  6.1K   11GB |  1.2M  5.5TB | 4.8TB |  17 17.7 |  44GB 152GB 448GB
 ---------------------------------------------------------------------------------------------------------------------------------------
 WAL: 1 files (3.8MB)  in: 315GB  written: 316GB (0% overhead)
 Flushes: 89210
 Compactions: 158752  estimated debt: 138GB  in progress: 1 (195MB)
              default: 152661  delete: 0  elision: 0  move: 6091  read: 0  rewrite: 0  multi-level: 7590
 MemTables: 1 (4.0MB)  zombie: 5 (20MB)
 Zombie tables: 8469 (155GB)
 Backing tables: 0 (0B)
 Virtual tables: 0 (0B)
 Block cache: 0 entries (0B)  hit rate: 0.0%
 Table cache: 1.9K entries (1.5MB)  hit rate: 87.1%
 Secondary cache: 0 entries (0B)  hit rate: 0.0%
 Snapshots: 0  earliest seq num: 0
 Table iters: 43
 Filter utility: 0.0%
 Ingestions: 0  as flushable: 0 (0B in 0 tables)

So things are going... slow... and there is probably a lot of contention to write now because it is compacting tables into the lower levels. That said, this node has HDDs and they were very slow before as well, so it is not conclusive.

Going forward I would like to setup another node with flatfs (blocks) + pebble (everything else).

Also, always worth remembering my notes here: #9678