[Relay Miner] Address high memory usage

[okdas]

Objective

RelayMiner processes relays but consumes a large amount of RAM, often leading to the process getting killed due to OOM if resource constrained. RelayMiner should be able to handle a larger amount of requests without saturating memory.

Origin Document

As a part of our load testing plan on TestNet, we've discovered this behavior as we send larger amounts of relays than we usually do on smaller networks.

Pprof heap snapshot suggests most of the memory is reserved by BadgerDB:

Goals

• Improve memory management to prevent OOM kills.
• Ensure RelayMiner can handle a larger volume of relays efficiently.
• Optimize or replace BadgerDB usage to reduce memory footprint.
• Related to this SMT ticket: [Scalability Design Document] Design document and plan to test SMT scalability smt#23

Deliverables

• Analyze memory usage hotspots within BadgerDB and optimize configurations.
• Investigate and implement code optimizations for better memory management.
• Test optimizations under load conditions to ensure stability and performance.
• Document changes and best practices for memory management.
• If necessary, evaluate and replace BadgerDB with a more memory-efficient alternative.

Non-goals / Non-deliverables

• Changes to the core RelayMiner functionality beyond memory management.
• Major refactoring of unrelated RelayMiner components.

General deliverables

• Testing: Add new tests (unit and/or E2E) to the test suite.
• Documentation: Update architectural or development READMEs.

---

Creator: @okdas
Co-Owners: TBD

[okdas]

Synched with @Olshansk on this - I'll do a first pass to investigate if there are low-hanging fruits in badger configuration before we dig deeper into the code/smt changes.

[Olshansk]

Things to consider:

• Is there to avoid loading the whole key-value store into memory? Are we even doing this at all?
• What would the usage be if we were to handle "serious traffic"?
• Why isn't this a problem in Morse?
• Is using LevelDB, RocksDB, BoltDB or other better?

[okdas]

With PRs merged, the current resource utilization seems adequate. There's more room for improvements and we can work on that when needed.