diffRewardMap error for test that transfers funds from treasury to stake address using MIR certificate

[ArturWieczorek]

OS:
Your OS: Linux Mint 20.2 Cinnamon
Linux Kernel: 5.4.0-90-generic

Machine Specs:
Processor: AMD Ryzen 9 3900X 12-Core Processor × 12
RAM: 32 GB

Versions
The db-sync version : 32e69cd7010686bc7438473e4c4c48511058a31d on master

PostgreSQL version: 12.8

Build/Install Method
The method you use to build or install cardano-db-sync:
cardano-db-syncwas built with nix using official instruction docs.

Run method
The method you used to run cardano-db-sync (eg Nix/Docker/systemd/none):

PGPASSFILE=config/pgpass db-sync-node/bin/cardano-db-sync --config config/config.yaml --socket-path ../cardano-node/state-node/node.socket --schema-dir schema/ --state-dir ledger-state/qa_cluster

Additional context
This was discovered during our nighlty tests. Error happened during a test that transfers funds from treasury to stake address using MIR certificate. This was for commit: 32e69cd on master however the validate.hs https://github.com/input-output-hk/cardano-db-sync/blob/master/cardano-db-sync/src/Cardano/DbSync/Era/Shelley/Validate.hs has not been changed in 20 days so this can happen for version 12.0.0 of db-sync too.

The test: https://github.com/input-output-hk/cardano-node-tests/blob/master/cardano_node_tests/tests/test_governance.py#L602

Problem Report

diffRewardMap error for test that transfers funds from treasury to stake address using MIR certificate.

[db-sync-node:Error:1048] [2021-12-11 03:18:24.61 UTC] diffRewardMap:
[db-sync-node:Error:1048] [2021-12-11 03:18:24.61 UTC]   e011b1685a333acb205662f6f21a89668c21b3cb189e5ffc4b647e6c63: [(RwdTreasury,DbLovelace 50000000),(RwdTreasury,DbLovelace 50000000)] /= [(RwdTreasury,Coin 50000000)]

[erikd]

@ArturWieczorek What era does that transfer happen in? Are you sure there were not two transfers in that epoch?

[ArturWieczorek]

@erikd It happens in Alonzo era.

This is one test https://github.com/input-output-hk/cardano-node-tests/blob/master/cardano_node_tests/tests/test_mir_certs.py#L429 but it is parametrized:

@pytest.mark.parametrize("fund_src", (RESERVES, TREASURY))

so in effect there are two test "instances", one sends funds from the reserves pot to stake address and the other one from treasury pot to stake address.

I asked about two transfers @mkoura and have some details from him:

Because of the locking mechanism tests cannot run at the same time on single cluster instance. It cannot run in single epoch either, because in one epoch the MIR cert is submitted, then the test waits for another epoch and checks the transfer. So the MIR certs cannot be submitted in the same epoch. Every MIR tests wait for another epoch to check the effect, so it cannot be side effect of other test either.