TheGraph Node Indexing Error

[AlfredoG87]

Description

When indexing a sub-graph with synthetic transactions, from time to time the following logs appears on the graph-node console.

INFO Trying again after block polling failed: Receipt for tx 0x965fc6051ad8302ab908841457e0acbdcc897e2aaf0f9432c028ac2641e85367 unavailable, block was likely uncled (block hash = 0x2cd5b2c2f091d24d1204b1ec59996a20fe6f84bc16d4ceca27f94be4fee08217), provider: mainnet-rpc-0, component: BlockIngestor"

The logs are similar but change on tx hash and block hash however when I do the manual lookup of the transaction receipt:

curl --location 'https://mainnet.hashio.io/api' \
--header 'Content-Type: application/json' \
--data '{
    "method": "eth_getTransactionReceipt",
    "params": [
       "0x965fc6051ad8302ab908841457e0acbdcc897e2aaf0f9432c028ac2641e85367"
    ],
    "id": 68,
    "jsonrpc": "2.0"
}'

or the block:

curl --location 'https://mainnet.hashio.io/api' \
--header 'Content-Type: application/json' \
--data '{
    "jsonrpc": "2.0",
    "method": "eth_getBlockByHash",
    "params": [
                "0x2cd5b2c2f091d24d1204b1ec59996a20fe6f84bc16d4ceca27f94be4fee08217",
                true
    ],
    "id": 1
}'

Everything seems as expected.

Steps to reproduce

Run a long term sub-graph for a HTS Token that has movement, in this case I chose SAUCE Token.

Additional context

No response

Hedera network

mainnet

Version

v0.29.0

Operating system

None

[AlfredoG87]

Today I found a persistent similar issue that is not related to Synthetic transactions.
LOG:

Aug 10 16:12:56.944 INFO Trying again after block polling failed: Receipt for tx 0x8ae5216811e6968bea91020dccaeccf017fb2f603fc9810ba80d503e507a1d7a unavailable, block was likely uncled (block hash = 0xb13ba41b06bce1d637bec7e64f44a39c6fbc263864c884cb808e909d77e9bc2b), provider: mainnet-rpc-0, component: BlockIngestor

when querying the block by hash, there is the transaction:

curl --location 'https://mainnet.hashio.io/api' \
--header 'Content-Type: application/json' \
--data '{
    "jsonrpc": "2.0",
    "method": "eth_getBlockByHash",
    "params": [
                "0xb13ba41b06bce1d637bec7e64f44a39c6fbc263864c884cb808e909d77e9bc2b",
                true
    ],
    "id": 1
}'

Response:

{
    "result": {
        "timestamp": "0x64d50b30",
        "difficulty": "0x0",
        "extraData": "0x",
        "gasLimit": "0xe4e1c0",
        "baseFeePerGas": "0x1484a8c9400",
        "gasUsed": "0xfa000",
        "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
        "miner": "0x0000000000000000000000000000000000000000",
        "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "nonce": "0x0000000000000000",
        "receiptsRoot": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
        "size": "0xa3275",
        "stateRoot": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "totalDifficulty": "0x0",
        "transactions": [
            {
                "blockHash": "0xb13ba41b06bce1d637bec7e64f44a39c6fbc263864c884cb808e909d77e9bc2b",
                "blockNumber": "0x316eab2",
                "chainId": null,
                "from": "0x000000000000000000000000000000000019df18",
                "gas": "0xfa000",
                "gasPrice": null,
                "hash": "0xb1b76937b109b72a1b3595f770a436198c06f2e8e29d0eb96c74189637c9b8c3",
                "input": "0x7ff36ab500000000000000000000000000000000000000000000000000002cb46cfe2a040000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000019df1800000000000000000000000000000000000000000000000000000189e0457f8f00000000000000000000000000000000000000000000000000000000000000030000000000000000000000000000000000000000000000000000000000163b5a00000000000000000000000000000000000000000000000000000000000b2ad50000000000000000000000000000000000000000000000000000000000307911",
                "maxPriorityFeePerGas": null,
                "maxFeePerGas": null,
                "nonce": "0x0",
                "r": null,
                "s": null,
                "to": "0x00000000000000000000000000000000002e7a5d",
                "transactionIndex": "0x1cf",
                "type": null,
                "v": "0x0",
                "value": "0x4a817c800"
            },
            {
                "blockHash": "0xb13ba41b06bce1d637bec7e64f44a39c6fbc263864c884cb808e909d77e9bc2b",
                "blockNumber": "0x316eab2",
                "chainId": null,
                "from": "0x000000000000000000000000000000000019df18",
                "gas": "0x0",
                "gasPrice": null,
                "hash": "0x8ae5216811e6968bea91020dccaeccf017fb2f603fc9810ba80d503e507a1d7a",
                "input": "0x7ff36ab500000000000000000000000000000000000000000000000000002cb46cfe2a040000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000019df1800000000000000000000000000000000000000000000000000000189e0457f8f00000000000000000000000000000000000000000000000000000000000000030000000000000000000000000000000000000000000000000000000000163b5a00000000000000000000000000000000000000000000000000000000000b2ad50000000000000000000000000000000000000000000000000000000000307911",
                "maxPriorityFeePerGas": null,
                "maxFeePerGas": null,
                "nonce": "0x0",
                "r": null,
                "s": null,
                "to": "0x00000000000000000000000000000000002e7a5d",
                "transactionIndex": "0x61f",
                "type": null,
                "v": "0x0",
                "value": "0x4a817c800"
            }
        ],
        "transactionsRoot": "0xb13ba41b06bce1d637bec7e64f44a39c6fbc263864c884cb808e909d77e9bc2b",
        "uncles": [],
        "number": "0x316eab2",
        "hash": "0xb13ba41b06bce1d637bec7e64f44a39c6fbc263864c884cb808e909d77e9bc2b",
        "parentHash": "0x4ab3958b537a8e2578cfa5be68ee79ae4f4f6ca66d4ba9a96ee951fe6b47a62e"
    },
    "jsonrpc": "2.0",
    "id": 1
}

The transaction does not seem synthetic, due to the value.

however when trying to get the tx receipt:

curl --location 'https://mainnet.hashio.io/api' \
--header 'Content-Type: application/json' \
--data '{
    "method": "eth_getTransactionReceipt",
    "params": [
       "0x8ae5216811e6968bea91020dccaeccf017fb2f603fc9810ba80d503e507a1d7a"
    ],
    "id": 68,
    "jsonrpc": "2.0"
}'

Response:

{
    "result": null,
    "jsonrpc": "2.0",
    "id": 68
}

[AlfredoG87]

@georgi-l95 I was able to confirm that this issue is happening due to the lack of Distributed Cache implementing IP Affinity at the LB could mitigate it while we resolve the root cause.