Reward accounting issue on mainnet

[erikd]

Syncing from scratch on mainnet with master plus @kderme 's fix in #946 I get:

[2021-11-12 04:09:01.21 UTC] Starting epoch 213
[2021-11-12 04:09:01.27 UTC] Handling 35360 stakes for epoch 213 slot 6652800, hash 8839b697618eb1b3167bcd2658e10008d9c1d11bd32b305abf497371cd79dafa
[2021-11-12 04:09:03.18 UTC] validateEpochRewards: rewards spendable in epoch 213 expected total of 7235499.325914 ADA but got 7235489.131814 ADA
[2021-11-12 04:09:03.31 UTC] diffRewardMap:
[2021-11-12 04:09:03.31 UTC]   e1001ec42376196df0b85b6ad660e7419e77b51b16fda8b931d559562e: [(RwdLeader,DbLovelace 643870191)] /= [(RwdLeader,Coin 643870191),(RwdMember,Coin 971)]
[2021-11-12 04:09:03.31 UTC]   e101fad190adc72e0884adb77aa72ef9d95d3adb75f782bad63b53a5b7: [(RwdLeader,DbLovelace 1197512732)] /= [(RwdLeader,Coin 1197512732),(RwdMember,Coin 1437)]
[2021-11-12 04:09:03.31 UTC]   e124f1b3791037e55363815fc4b788b886cd1d82a2f2d39e17196cd1a4: [(RwdLeader,DbLovelace 431236474)] /= [(RwdLeader,Coin 431236474),(RwdMember,Coin 1480)]
[2021-11-12 04:09:03.31 UTC]   e146679d60511973cd756322e6fdd1b0d6224eebec681f065a704e753e: [(RwdLeader,DbLovelace 821873524)] /= [(RwdLeader,Coin 821873524),(RwdMember,Coin 364746)]
[2021-11-12 04:09:03.31 UTC]   e146a31eb43ac7d42255b74f24945badfc5da0014a2e37e6bcb69ce4d9: [(RwdLeader,DbLovelace 455667597)] /= [(RwdLeader,Coin 455667597),(RwdMember,Coin 2662)]
[2021-11-12 04:09:03.31 UTC]   e1854139fc8987990fd89699beb1b59b09c047ace356870dcaadc93b22: [(RwdLeader,DbLovelace 1679331533)] /= [(RwdLeader,Coin 1679331533),(RwdMember,Coin 6401)]
[2021-11-12 04:09:03.31 UTC]   e1aaab46ce0889442fe83e74489970ef6337721fe8761478825ec7701e: [(RwdLeader,DbLovelace 687065251)] /= [(RwdLeader,Coin 687065251),(RwdMember,Coin 9546263)]
[2021-11-12 04:09:03.31 UTC]   e1db0de66e9113e8ddecc3642124a557fe0d7753d642162cc1ec85292c: [(RwdLeader,DbLovelace 369093412)] /= [(RwdLeader,Coin 369093412),(RwdMember,Coin 270140)]
panic abort:
Rewards differ between ledger and db-sync.

For each row we have the stake address, then the list of rewards for that address from the database (ie DbLovelace) and then the rewards from the ledger.

[erikd]

Interestingly if I take out the special handling for the Shelley epoch then there is no validation failure.

This looks like its a regression in ledger-specs .