Testnet rewards differences

[sorki]

Current master with rewards validation would panic on two epochs w/o patch. Managed to capture diffs as well this time. Not a big deal currently but it might bite some people.

Epoch 84

[db-sync-node:Warning:81] [2021-09-09 17:37:30.48 UTC] validateEpochRewards: rewards earned in epoch 84 expected total of 158768.12877 ADA but got 158468.12877 ADA
dbMap length: 27 
ledgerMap length: 27 
diffMap length: 3
e03412de65e523666ef5df0a7c3db510b99b0553b67f324f0ba215af82 (118244, 1): [(RwdMember,DbLovelace 118244)], Coin 100118244
e07821cc41d016e17e2f6e37a8c1ba8a3caaaf52d44b80a557b4cc4d98 (128667, 1): [(RwdMember,DbLovelace 128667)], Coin 100128667
e09cd7cd070bbcc65f75b460a3f4462e3445f11a7f2f1a49825a10f88d (128667, 1): [(RwdMember,DbLovelace 128667)], Coin 100128667
Rewards differ between ledger and db-sync.
Please report at https://github.com/input-output-hk/cardano-db-sync/issues.
XXX: Would panic on epochNo EpochNo 84 

Epoch 107

[db-sync-node:Warning:81] [2021-09-09 18:05:46.39 UTC] validateEpochRewards: rewards earned in epoch 107 expected total of 523029.532638 ADA but got 523129.532638 ADA
dbMap length: 239
ledgerMap length: 238
diffMap length: 1
e0797c8091fcd3591aaa23285abf0e29af1432dd77b6d02c046f7f9fbb (100000000, 1): [(RwdTreasury,DbLovelace 100000000)], Coin (-1)
Rewards differ between ledger and db-sync.
Please report at https://github.com/input-output-hk/cardano-db-sync/issues.
XXX: Would panic on epochNo EpochNo 107

[erikd]

This is almost certainly related to #796 .

[ArturWieczorek]

@erikd I see errors similar to that one on testnet when using:

cardano-node: tag: 1.31.0-rc1 commit c7d0f38e73fafc4d9e0e7bfbac62ca968f99141a
cardano-db-sync: tag: 12.0.0-pre3 commit 6e59465

for epochs = [86-88, 109, 138-165] (165 being the latest epoch at the moment of running db-sync )

Log examples:

[db-sync-node:Info:59] [2021-10-27 11:27:25.97 UTC] Starting epoch 86
[db-sync-node:Info:59] [2021-10-27 11:27:25.98 UTC] Handling 256 stakes for epoch 86 slot 6782400, hash 3fdfbd1bed0358d16778ae4d0bd5e46212cff3d1a9a155a1fb8574fbb7f10497
[db-sync-node:Warning:59] [2021-10-27 11:27:26.01 UTC] validateEpochRewards: rewards spendable in epoch 86 expected total of 158768.12877 ADA but got 158468.12877 ADA
[db-sync-node:Error:59] [2021-10-27 11:27:26.01 UTC] Rewards differ between ledger and db-sync.
Please report at https://github.com/input-output-hk/cardano-db-sync/issues.

[db-sync-node:Info:59] [2021-10-27 11:55:38.97 UTC] Starting epoch 143
[db-sync-node:Info:59] [2021-10-27 11:55:38.98 UTC] Handling 1980 stakes for epoch 143 slot 31406422, hash 6234e1e59a77ee0cddb1d83d74dfbc4fb7664ae74768207be9b6d60a1147b8cd
[db-sync-node:Info:59] [2021-10-27 11:55:39.14 UTC] validateEpochRewards: total rewards that become spendable in epoch 143 is 1331858.624017 ADA
[db-sync-node:Error:59] [2021-10-27 11:55:39.28 UTC] 3 errors, eg
  stake_test1uq6p9hn9u53kvmh4mu98c0d4zzuekp2nkelnynct5g26lqs9yenqu rewards are 1442.26932 ADA and withdrawals are 1542.26932 ADA

  stake_test1upuzrnzp6qtwzl30dcm63sd63g724t6j639cpf2hknxymxqnt30fu rewards are 1442.453622 ADA and withdrawals are 1542.453622 ADA

  stake_test1uzwd0ng8pw7vvhm4k3s28azx9c6ytug60uh35jvztgg03rge58jf8 rewards are 1442.453622 ADA and withdrawals are 1542.453622 ADA


[db-sync-node:Error:59] [2021-10-27 11:55:39.28 UTC] validateRewardWithdrawals: 143

Full log from testnet:
testnet_log.txt

[ArturWieczorek]

For cardano-node:

[cardano.node.basicInfo.version:Notice:5] [2021-11-17 08:44:14.32 UTC] 1.31.0
[cardano.node.basicInfo.commit:Notice:5] [2021-11-17 08:44:14.32 UTC] 2cbe363874d0261bc62f52185cf23ed492cf4859

and cardano-db-sync :

db-sync-node-extended/bin/cardano-db-sync-extended --version
cardano-db-sync-extended 12.0.0 - linux-x86_64 - ghc-8.10
git revision 2ee31a556c8293be8413565cd30ef5483d01554e
tag: 12.0.0-pre5

there are reward validation errors for:
epochs = [86-88, 138-169] (169 being the latest epoch at the moment of running db-sync )

Logs:
db_sync_12_pre5_testnet_log.txt

[kderme]

[db-sync-node:Info:59] [2021-10-27 11:55:39.14 UTC] validateEpochRewards: total rewards that become spendable in epoch 143 is 1331858.624017 ADA
[db-sync-node:Error:59] [2021-10-27 11:55:39.28 UTC] 3 errors, eg
  stake_test1uq6p9hn9u53kvmh4mu98c0d4zzuekp2nkelnynct5g26lqs9yenqu rewards are 1442.26932 ADA and withdrawals are 1542.26932 ADA

  stake_test1upuzrnzp6qtwzl30dcm63sd63g724t6j639cpf2hknxymxqnt30fu rewards are 1442.453622 ADA and withdrawals are 1542.453622 ADA

  stake_test1uzwd0ng8pw7vvhm4k3s28azx9c6ytug60uh35jvztgg03rge58jf8 rewards are 1442.453622 ADA and withdrawals are 1542.453622 ADA

For all these addresses there are instant rewards from treasury that are missing from the reward table.