ClusterSlots uses too much memory

[sakridge]

Problem

const CLUSTER_SLOTS_TRIM_SIZE: usize = 524_288; // 512K
pub type SlotPubkeys = HashMap<Pubkey, u64>;

#[derive(Default)]
pub struct ClusterSlots {
    cluster_slots: RwLock<BTreeMap<Slot, Arc<RwLock<SlotPubkeys>>>>,

hashbrown::raw::RawTableInner$LT$A$GT$::prepare_resize::hd9de4d7705dc4632
  in /home/stephen_solana_com/solana/target/release/solana-validator
5.89GB consumed over 1807948 calls from:
    0x55b0a435cb61
      in /home/stephen_solana_com/solana/target/release/solana-validator
    _$LT$hashbrown..map..HashMap$LT$K$C$V$C$S$C$A$GT$$u20$as$u20$core..iter..traits..collect..Extend$LT$$LP$K$C$V$RP$$GT$$GT$::extend::hc3281369a9ad10d2
      in /home/stephen_solana_com/solana/target/release/solana-validator
    solana_core::cluster_slots::ClusterSlots::update::h8fcd014c2713807e
      in /home/stephen_solana_com/solana/target/release/solana-validator
    solana_core::cluster_slots_service::ClusterSlotsService::run::h4f4d6ebf13410164
      in /home/stephen_solana_com/solana/target/release/solana-validator
    std::sys_common::backtrace::__rust_begin_short_backtrace::h253287d3ac949d1d
      in /home/stephen_solana_com/solana/target/release/solana-validator
    core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h5d585c3e7664364f
      in /home/stephen_solana_com/solana/target/release/solana-validator
    _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::hc444a77f8dd8d825
      at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/alloc/src/boxed.rs:1546
      in /home/stephen_solana_com/solana/target/release/solana-validator
    _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h8b68a0a9a2093dfc
      at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/alloc/src/boxed.rs:1546
    std::sys::unix::thread::Thread::new::thread_start::hb95464447f61f48d
      at library/std/src/sys/unix/thread.rs:71
    start_thread
      in /lib/x86_64-linux-gnu/libpthread.so.0
    __clone
      in /lib/x86_64-linux-gnu/libc.so.6

Proposed Solution

Can we reduce CLUSTER_SLOTS_TRIM_SIZE ?

Or find a more efficient way to store this information. Instead use a map of pubkey -> BitVec or compressed BitVec?

Page out to disk in files or blockstore?

Consumers:
replay stage -> propagation stats
repair service -> pruning slots

[sakridge]

@carllin @behzadnouri What do you guys think?

[behzadnouri]

Yeah, let me look into this, since it was brought up in #14366 (comment) as well.

CLUSTER_SLOTS_TRIM_SIZE doesn't do much since it only limits the size of the outer map. Inner ones still can get pretty large. It is only intended to mitigate the case where a node receives bogus epoch slots.

But, bit-vec idea is promising. I will give it a shot.

[carllin]

Yeah I was also thinking maybe you could probably page out:

1. Much smaller slots
2. Validators with low/0 stake as they probably aren't used for propagation stats, only rarely if they are selected for repair (Could represent all the 0 staked validators with one representative from that 0 staked set and occasionally shuffle that representative for instance).

[behzadnouri]

Testing on TDS seems like dropping unstaked nodes does not help much. Cluster slots still has ~520k inner hashmaps. So the slots are spread over a range of size ~520k which sounds very suspicious to me.
Some snapshot of the slots with highest number of staked nodes look like:

[(80435455, 1755), (80435451, 1752), (80435450, 1752), (80435467, 1747), (80435452, 1747), (80435454, 1744), (80435475, 1743), (80435463, 1743), (80435471, 1739), (80435470, 1739), ....]

and with lowest number of staked nodes (from the same snapshot):

[(80556216, 1), (80556212, 1), (80556211, 1), (80556210, 1), (80556209, 1), (80556208, 1), (80556207, 1), (80556206, 1), (80556205, 1), (80556204, 1), ...]

[behzadnouri]

This seems to be because of crds values from nodes with different shred_version are creeping in. Excluding them from get_epoch_slots will fix this. I will send out a patch soon.

[behzadnouri]

reopening this since cluser_slots is still consuming 7GB on mainnet:
https://discord.com/channels/428295358100013066/439194979856809985/865513541820743720

[behzadnouri]

Checking on testnet, all the erroneous epoch-slots are from: 77VrLXNcVnpPHcMCcfkNU8SpuPzoCbGfp3Kqc2iV1dgu
which is using same identity for both mainnet and testnet; and so causing cross contamination between clusters.

Probably the node does not halt because NodeInstances are not propagated due to different shred-versions:
https://github.com/solana-labs/solana/blob/faf99f476/gossip/src/cluster_info.rs#L2321-L2326

However EpochSlots are cross-contaminating which is confusing why. Maybe ContactInfo is updated after shred-version of EpochSlots is checked and so it bypasses the shred-version check. But not sure why the same does not happen with NodeInstance.

On testnet:

Identity                                      Vote Account                            Commission  Last Vote       Root Slot    Skip Rate  Credits  Version  Active Stake
77VrLXNcVnpPHcMCcfkNU8SpuPzoCbGfp3Kqc2iV1dgu  213LE1Jpb4KwfAeqnWY9dzr3TpYoadGk2oQmHXML2AdZ  100%  88898340 ( -2)  88898293 ( -1) 100.00%   124619    1.7.9  5000.98401984 SOL (0.01%)

IP Address        |Age(ms)| Node identifier                              | Version |Gossip| TPU  |TPUfwd| TVU  |TVUfwd|Repair|ServeR|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+------+------+------+------+------+--------
157.90.210.250    |   338 | 77VrLXNcVnpPHcMCcfkNU8SpuPzoCbGfp3Kqc2iV1dgu |  1.7.9  | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 18122

On mainnet:

Identity                                      Vote Account                            Commission  Last Vote       Root Slot    Skip Rate  Credits  Version  Active Stake
77VrLXNcVnpPHcMCcfkNU8SpuPzoCbGfp3Kqc2iV1dgu  8RCqWaeUgVvPEuHxa3SchprM1jm2dyJTbEYGmfMBmiyQ   10%  91087005 ( -2)  91086940 ( -4)      -    269272   1.6.20  200.908629969 SOL (0.00%)

IP Address        |Age(ms)| Node identifier                              | Version |Gossip| TPU  |TPUfwd| TVU  |TVUfwd|Repair|ServeR|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+------+------+------+------+------+--------
65.21.94.24       |   260 | 77VrLXNcVnpPHcMCcfkNU8SpuPzoCbGfp3Kqc2iV1dgu | 1.6.20  | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 13490

[behzadnouri]

#19190 should mitigate above situation where a node is using same identity key for testnet and mainnet clusters. Also, announced on discord for the nodes to change their identity keys:
https://discord.com/channels/428295358100013066/594138785558691840/874801862853939200