Encountering a consistent indexing error on Matic "Found no transaction for event"

[TravelingTechGuy]

1. This is a bug report
2. I'm trying to deploy my subgraph. It used the ABI of the USDT contract, and is supposed to index an entity whenever the to in the Transfer event is the address of our contract (i.e. whenever someone staked USDT in the contract).

This works well on Ethereum. On Matic (Polygon), we get the following fatal error:

"failed to process trigger: block #15167232 (0x7b2a…0d6b), transaction 4641fb604c8b6b9d515f57f364e8039fb600ed0471a7dcb7f16f2bd1314501f1: Found no transaction for event"

This is consistent, and reproducible. I tried adding a function that leaves the handler in case: the block number is 15167232, the block number is between 15167200 and 15167240 (thought a range would help), if event.transaction is empty - to no avail, this error still happens. My guess is this happens before my handler code, and therefore, not under my control.

My workaround is to add a higher startBlock number to the yaml, but the we lose tons of events.

My expected behavior is either a resolution that allows me to index the graph, or a way to bypass whatever problem triggers the fatal error.

More details:

1. The subgraph can be found at: https://thegraph.com/explorer/subgraph/travelingtechguy/cvipolygon
2. I can provide a link to the code, if needed. It has other entities that index well.

[TravelingTechGuy]

Update: I advanced the startBlock for the USDT contract past block 15167232, but it just blew up later with the same error:

failed to process trigger: block #15326592 (0x9641…dfa9), transaction 626d7374ca1862fe801fad8b57418fddd19f826b2d9797a014c5589da343985d: Found no transaction for event"

[azf20]

Thanks for reporting @TravelingTechGuy - can you please share a link to the code so we can replicate?
This looks like the same issue as #2353, but on a different network (Matic, not Celo) cc @tilacog and @leoyvens

[TravelingTechGuy]

Hi @azf20 - thanks for looking into this!

If you look at this branch - the USDT2.json is the USDT ABI I took from the Matic explorer. Change the startBlock in the yaml to a number below 15167232 and run the deployment. You can completely discard the other CVI entities.

In fact, we've now separated them from the USDT contract, as they index well, and USDT blows up.

In short, I'd say the quickest recreation steps:

1. Create a new subgraph
2. Grab the UST ABI from the explorer
3. Index the Transfer event
4. Start at startBlock close to one of the error ones.
5. Deploy

[azf20]

We have replicated this, it is possibly related to the fact that the from and to address for the transaction are both 0x0000000000000000000000000000000000000000. Looking into it. Thanks for reporting!

e.g. here https://polygonscan.com/tx/0x626d7374ca1862fe801fad8b57418fddd19f826b2d9797a014c5589da343985d

[TravelingTechGuy]

Thanks for being on top of it @azf20!
I assume there's nothing I can do in my code to get around this, and that the error occurs prior to the handler being called?

I also wonder: a 0x to 0x transfer: it's not a burn per se - is that an indexing error, or some bot doing something?

[azf20]

It is a special case on Polygon, I think it relates to their bridge - the tokens aren't transferred from 0x to 0x, but the transaction itself is a self from 0x to 0x. We have a couple more instances of this, so hopefully we'll be able to isolate the issue

[TravelingTechGuy]

Thanks for the update @azf20 - i assumed it was something to do with Polygon, since we haven't run into it on mainnet.
In the meantime, is there anything I can do to guard against this in my code, or is it something that gets thrown before my code (handler) is even executed (i.e. I have no control of it)?

[azf20]

Yes I think this is throwing before the handler is initiated - @tilacog is investigating!

[azf20]

We have identified the issue, which relates to our Matic RPC not including all transactions in eth_getBlockByHash at some point, which was then cached in the database, resulting in the ongoing issue. This may relate to the new Bor requirement to specifically return bor logs. We are planning to clear the cache which should resolve the issue, as Matic RPCs are now returning the correct number of transactions for these blocks cc @leoyvens @tilacog

[azf20]

We cleared the cache, which has resolved this issue for the subgraph we were testing with - @TravelingTechGuy you should be able to restart this subgraph before the problematic block (15167232), and it should index successfully. I will close this issue, but please do re-open if you see a recurrence!

[azf20]

This recurred on the Decentraland subgraph:

Subgraph instance failed to run: failed to process trigger: block #16574080 (0x502d…2b95), transaction 384650eb47fd17640598f72eb3eeff356ab16a02ec3c5c33d3de818763a29081: Found no transaction for event, code: SubgraphSyncingFailure

[tilacog]

Since this error is solved by re-fetching the block, would it make sense to implement some retry mechanism whenever this error surfaces?

[miguelmota]

Started encountering this error as well and I'm not sure how to get around it. Any help is greatly appreciated

$curl -s --location --request POST https://api.thegraph.com/index-node/graphql --data-raw '{"query":"{ indexingStatusForPendingVersion(subgraphName: \"hop-protocol/hop-polygon\") { subgraph fatalError { message } nonFatalErrors {message } } }"}'
{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16573056 (0x00d5…7cad), transaction 059189fea36971157989bf3c13b17743c9313df88f317b3fa68d59793c117c03: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmWEF3CEusZDBLwkM2ijUoFH8yuFbaFZcE2EAUpr8ykxbk"}}}

[leoyvens]

@tilacog The question is when do we know if we need to retry. Maybe if we double check the array of transactions received against eth_getTransactionCount?

[tilacog]

@miguelmota We've re-fetched the matic faulty block at the hosted service. Can you please confirm if the error is gone now?

[miguelmota]

@tilacog A subgraph reedeployment was made but the issue occurred again towards the end of the sync

https://thegraph.com/legacy-explorer/subgraph/hop-protocol/hop-polygon?version=pending

[tilacog]

@miguelmota Thanks for the extra info.
I've re-fetched three faulty blocks from the hosted service, and this subgraph resumed syncing.

Please let us know if this incident happens again, and we'll try to fix the problem.

[miguelmota]

@tilacog looks to be fixed! Thanks!

[miguelmota]

@tilacog There's another failure here
https://thegraph.com/legacy-explorer/subgraph/hop-protocol/hop-polygon?version=pending

[tilacog]

@miguelmota
the subgraph has resumed syncing.

Thanks for the heads up!

[reddyismav]

@leoyvens did you happen to try what you suggested , eth_getTransactionCount and then comparing it with the array returned from getBlockByHash. ?

[reddyismav]

In the meanwhile m trying to figure out why the sync issue on the bor nodes is happening and why is it aways for state sync transactions.
I ll keep you guys updated here.

[miguelmota]

@tilacog thanks again!

[miguelmota]

Sync error - block #16785216

{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16785216 (0xdc6c…85c5), transaction 8d63b25cbc8e3fa4a62b4513d70151f8485c1fe1d5da044a83789a47219a6b55: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmfDg1oHAPb7TPuy882BfBFZrJKMPNEuib98Em7rFds2nb"}}}

@tilacog

[miguelmota]

Sync error - block #16797184

{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16797184 (0x1d93…909a), transaction df952e2b5bb59beb5acbced5be788277312ec838142a808fcdbe437cff8ff12f: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmfDg1oHAPb7TPuy882BfBFZrJKMPNEuib98Em7rFds2nb"}}}

@tilacog

[shanefontaine]

Hey @tilacog

Messages on Polygon that are sent from L1 appear as events that are emitted from the 0x0 address. For example, here is a "transaction" that has an event from one of our users that sent a message from L1 -> L2. This message emitted an event (topic 0x320958176930804eb66c2343c7343fc0367dc16249590c0f195783bee199d094) on L2 that you can see in this transaction.

On any other chain, I would expect this event to be emitted on the transaction that actually updated state. However, on Polygon, the transaction that emits the event is not the transaction that updates state. Might this be the issue that the indexer is running into?

[miguelmota]

Sync error - block #16800192

{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16800192 (0x131d…5b30), transaction 18ad7854383bfb6bf2712aa95b710ffc254ce91e882c3ed6306f929322dcdd53: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmfDg1oHAPb7TPuy882BfBFZrJKMPNEuib98Em7rFds2nb"}}}

@tilacog

[schmidsi]

Hey @miguelmota and others here: We cleaned our db and apparently the new Bor version should fix our problem. Your subgraph is syncing. We hope that we now finally have a stable situation. Let us know if there are any problems.

[miguelmota]

Thanks for the update @schmidsi! Have not seen any syncing errors since the update 🎉

[schmidsi]

Nice. I close this issue then. Feel free to reopen if the problem still persists.

[schmidsi]

Seems like we have this problem now on Mumbai: https://thegraph.com/legacy-explorer/subgraph/lemonadesocial/lemonade-marketplace?version=pending

[schmidsi]

https://thegraph.com/legacy-explorer/subgraph/lemonadesocial/lemonade-marketplace?version=pending failed again with error: "failed to process trigger: block #18160704 (0xfa5e…0628), transaction ff72d1f7e52a138fa8ff82190215119118b7a66f16844eadca4d0e053cf2f6b2: Found no transaction for event"

[azf20]

Thanks. This is strange, as this was one of the blocks which was cleared from the cache, can you confirm @tilacog ?

[tilacog]

Yes, I can confirm this block was previously deleted.

[schmidsi]

Again 🤯: https://thegraph.com/legacy-explorer/subgraph/lemonadesocial/lemonade-marketplace?version=pending

failed to process trigger: block #18160704 (0xfa5e…0628), transaction ff72d1f7e52a138fa8ff82190215119118b7a66f16844eadca4d0e053cf2f6b2: Found no transaction for event

[ChrisLahaye]

@tilacog could you please help us out with https://thegraph.com/legacy-explorer/subgraph/lemonadesocial/lemonade-marketplace?version=pending

failed to process trigger: block #18223680 (0xd0aa…7fee), transaction 7651a8dca5c6ece8758e46aabf29daddc3fdd504dded7d6cde0f0c9f29ff5d26: Found no transaction for event

Thank you

[steve-ng]

hello @tilacog we noticed this on self-hosted graph node as well, is there an easy way to resync a block?

[tilacog]

Currently, to resync a faulty block you must delete the block record from its database table and restart graph-node afterwards.

[piavgh]

@tilacog : which table are you referring to?

[tilacog]

@piavgh It depends on how graph-node is configured.

It can be either the ethereum_blocks table in primary or the chainXX.blocks table in a dedicated shard.

The chainXX is the Postgres schema identifier for a given network and can be found by querying the public.chains table.

[piavgh]

@tilacog : thanks, I will check to see if I can delete the error block

Btw, why the ethereum_blocks table is so small? I'm indexing the Cronos chain (more than 1 million block), but this table is only about 8 bytes?

[tilacog]

@piavgh Then I believe your block cache should be in a dedicated table, as in the chainXX case I've mentioned above.

You can check your config file to discover where those blocks are being kept.
You can also select the table public.chains to discover the same.

[vietthang207]

@tilacog : thanks, I will check to see if I can delete the error block

Btw, why the ethereum_blocks table is so small? I'm indexing the Cronos chain (more than 1 million block), but this table is only about 8 bytes?

The actual table is in a different namespace. You can use \dn to list the namespace, and query the table in the other namespace.

[azf20]

Closing this for now, please re-open if you see this issue again on Polygon (Matic)

[OscBacon]

Hi, seems like Mumbai still has this issue, on block 18122880. The issue again is on a transaction from the zero address.
Note, I am using the hosted service

Subgraph failed with non-deterministic error: failed to process trigger: block #18122880 (0xcc82…42d3), transaction 553d80020c38eae2002b5063c264fe9b7c0b5c422859c5da9df221d40caa2f50: Found no transaction for event, retry_delay_s: 3060, attempt: 9

[gssvv]

Still experiencing this issue with zkSync Era mainnet:

Error: failed to process trigger: block #9376171 (0x01fa…404a), transaction 0000000000000000000000000000000000000000000000000000000000000000: Found no transaction for event

[incrypto32]

@gssvv can you share the deployment ID of the subgraph ? also that block doesnt seem to exist on zksync era

[gssvv]

@incrypto32 Sure, it's QmW5rbKe8iq98iJD2f2D4CRqkLecu3qZpbkH7hYFvTshkp

[KolevDarko]

I'm also facing the same thing issue, but was curious if this only happens when indexing tokens.
@gssvv are you indexing a token contract as well?

[xsteadybcgo]

The same issue with @gssvv , any workaround here?