TdS consensus lost at slot 797572

[mvines]

The TdS cluster lost consensus over the weekend.

• Last root slot: 797453
• Metrics window. No unusual blockstore errors.
• Looking at a ledger graph around time, it can be seen that the BSV forked left at 797572 and the rest of the stake forked right: s-798000.pdf

[mvines]

I wonder if this is really just the same issue as #8174

[mvines]

Bootstrap validator logs during this time window are at https://drive.google.com/drive/folders/1vj6uZuB5Lwn_bfWlPPMgg6WrnJTmGCI3?usp=sharing

Look for this line:

Feb  8 20:06:33 bootstrap-validator solana-validator[4674]: [2020-02-08T20:06:33.170791867Z INFO  solana_core::replay_stage] new fork:797572 parent:797571 (leader) root:797453

[carllin]

@aeyakovenko, @mvines @sagar-solana

I think a clearer picture off what happens is presented here:
forks.pdf starting at slot 797780. There are two forks:

1. Containing the bootstrap leader with id: 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on

2. All the other validators

These two forks run parallel to each other, seemingly without ever joining again

The reason for this seems to be the other validators are observing blocks from the bootstrap leader very late. A good example of this is slot 797891 on the bootstrap leader's fork.

This slot was frozen by the bootstrap leader here (search in logs above posted by @mvines)

[2020-02-08T20:10:32.118871763Z INFO  solana_runtime::bank] bank frozen: 797891 hash: J3Q93eahnu3LxChj3HBuabbtykAZCMi4UyMQVy33RdGJ accounts_delta: BankHash 6e8de403db81ca3107df96352b060f8fc069350d20db7c0e85464614b28949d4 signature_count: 0 last_blockhash: 2c2LF7P1Ujt4zperdQNjtmQCfortn1Kw7ww4YPh3HtnC

Meanwhile, in one of the validator's logs here: https://drive.google.com/open?id=1_O9LPuWWPwFwI_Yu6rLzGG_cDQ2jHDtV, it indicates they received the block almost 5 mins later

 [2020-02-08T20:15:43.830075155Z INFO  solana_runtime::bank] bank frozen: 797891 hash: J3Q93eahnu3LxChj3HBuabbtykAZCMi4UyMQVy33RdGJ accounts_delta: BankHash 6e8de403db81ca3107df96352b060f8fc069350d20db7c0e85464614b28949d4 signature_count: 0 last_blockhash: 2c2LF7P1Ujt4zperdQNjtmQCfortn1Kw7ww4YPh3HtnC

At the time this validator finally observed 797891 on fork #1, it also simultaneously observed a heavier block at 798418 on fork #2 (see below for the weight comparisons) so the validator doesn't switch:

On fork 1 at slot 797891

[2020-02-08T20:15:43.984559990Z WARN  solana_core::replay_stage] 2X5JSTLN9m2wm3ejCxfWRNMieuC2VMtaMWSoqLPbC4Pq slot_weight: 797891 5090111468178396653582698 1629253900651096942209625508730 797890

On fork 2 at slot 798418

[2020-02-08T20:15:43.986858761Z WARN  solana_core::replay_stage] 2X5JSTLN9m2wm3ejCxfWRNMieuC2VMtaMWSoqLPbC4Pq slot_weight: 798418 5090110939440567835796250 1629292802660799684221169001352 798417

[mvines]

@carllin - is there any action here still?

[carllin]

@mvines, yup, summarized by this issue: #8232

[mvines]

Obsoleted by #8232