Fatal: Merkle_mask.Masking_merkle_tree.Make(Inputs).Attached.Dangling_parent_reference on mainnet

[garethtdavies]

{"exn":{"commit_id":"a42bdeef6b0c15ee34616e4df76c882b0c5c7c2a","sexp":["monitor.ml.Error",["Merkle_mask.Masking_merkle_tree.Make(Inputs).Attached.Dangling_parent_reference(\"ec5df737-2e0c-3d42-b51a-e8df7b88af1a\", \"File \\\"src/lib/transition_router/transition_router.ml\\\", line 546, characters 61-68\")"],["Raised at file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 116, characters 10-57","Called from file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 531, characters 6-26","Called from file \"src/app/archive/archive_lib/diff.ml\", line 75, characters 17-57","Called from file \"src/list.ml\", line 336, characters 13-17","Called from file \"src/list.ml\" (inlined), line 361, characters 15-31","Called from file \"src/app/archive/archive_lib/diff.ml\", line 70, characters 6-420","Called from file \"src/lib/mina_lib/archive_client.ml\", line 73, characters 21-73","Called from file \"src/deferred_list.ml\", line 10, characters 19-26","Called from file \"src/deferred0.ml\", line 49, characters 2-10","Called from file \"src/lib/pipe_lib/broadcast_pipe.ml\", line 91, characters 25-28","Called from file \"src/pipe.ml\", line 827, characters 4-56","Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5","Called from file \"src/job_queue.ml\", line 171, characters 6-47","Caught by monitor coda"]],"backtrace":["Raised at file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 116, characters 10-57","Called from file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 531, characters 6-26","Called from file \"src/app/archive/archive_lib/diff.ml\", line 75, characters 17-57","Called from file \"src/list.ml\", line 336, characters 13-17","Called from file \"src/list.ml\" (inlined), line 361, characters 15-31","Called from file \"src/app/archive/archive_lib/diff.ml\", line 70, characters 6-420","Called from file \"src/lib/mina_lib/archive_client.ml\", line 73, characters 21-73","Called from file \"src/deferred_list.ml\", line 10, characters 19-26","Called from file \"src/deferred0.ml\", line 49, characters 2-10","Called from file \"src/lib/pipe_lib/broadcast_pipe.ml\", line 91, characters 25-28","Called from file \"src/pipe.ml\", line 827, characters 4-56","Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5","Called from file \"src/job_queue.ml\", line 171, characters 6-47"]},"host":"44.233.155.135","peer_id":"12D3KooWDwMibKmT86Hm6Eub7rbDDWfzv7wCgpoqw8g16qRmKe1r","pid":12,"port":8812}
  |  

[psteckler]

There's no monadic bind between getting the ledger and this failed call, so the mask was detached when obtained. Is there a guarantee of the attachment stability of a ledger in a breadcrumb?

The purpose of the call is to get a receipt chain status, which I believe is not meaningful now. Can we omit this code, somehow?

[nholland94]

This could be one of those rare race condition bugs we have seen because of the lack of transition frontier synchronization for extensions. This RFC specifies the issue and how to solve it, we just haven't had a chance to take this work yet. https://github.com/MinaProtocol/mina/blob/develop/rfcs/0028-frontier-synchronization.md

[deepthiskumar]

7 nodes on devnet crashed with this error, all around the same time
https://console.cloud.google.com/logs/viewer?interval=CUSTOM&project=o1labs-192920&minLogLevel=0&expandAll=false&timestamp=2021-05-18T04:32:59.408000000Z&customFacets=resource.labels.pod_name,jsonPayload.metadata.exn&limitCustomFacetWidth=true&advancedFilter=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22o1labs-192920%22%0Aresource.labels.location%3D%22us-east4%22%0Aresource.labels.cluster_name%3D%22coda-infra-east4%22%0Aresource.labels.namespace_name%3D%22devnet2%22%0Aresource.labels.container_name%3D%22coda%22%20OR%20resource.labels.container_name%3D%22coordinator%22%0A%22unhandled%20%22&scrollTimestamp=2021-05-18T04:17:59.213098668Z&dateRangeStart=2021-05-18T04:00:30.836Z&dateRangeEnd=2021-05-18T04:20:30.860Z

[EmreNOP]

minaprotocol/mina-daemon:1.2.0-fe51f1e-mainnet crash report
node crashed after block creation

Generating crash report
exn: {
"commit_id": "fe51f1ef5cfbcb4d5f559af39080963cf28ff1af",
"sexp": [
"monitor.ml.Error",
[
"Merkle_mask.Masking_merkle_tree.Make(Inputs).Attached.Dangling_parent_reference("f601d181-8062-3061-bdc0-91e56b866876", "File \"src/lib/mina_lib/mina_lib.ml\", line 1357, characters 49-56")"
],
[
"Raised at file "src/lib/merkle_mask/masking_merkle_tree.ml", line 116, characters 10-57",
"Called from file "src/lib/merkle_mask/masking_merkle_tree.ml", line 542, characters 6-26",
"Called from file "src/lib/network_pool/snark_pool.ml", line 265, characters 18-115",
"Called from file "src/lib/network_pool/snark_pool.ml", line 429, characters 10-59",
"Called from file "src/lib/network_pool/snark_pool_diff.ml", line 121, characters 17-65",
"Called from file "src/lib/network_pool/network_pool_base.ml", line 172, characters 22-68",
"Called from file "src/result.ml", line 151, characters 9-15",
"Caught by monitor coda"
]
],
"backtrace": [
"Raised at file "stdlib.ml", line 33, characters 17-33",
"Called from file "src/pipe.ml", line 873, characters 10-13",
"Called from file "src/job_queue.ml" (inlined), line 131, characters 2-5",
"Called from file "src/job_queue.ml", line 171, characters 6-47"
]
}
2021-11-03 23:15:33 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
exn: {
"commit_id": "fe51f1ef5cfbcb4d5f559af39080963cf28ff1af",
"sexp": [
"monitor.ml.Error",
[
"Failure",
"traced task R&snark_pool_verification failed, exception reported to parent monitor"
],
[
"Raised at file "stdlib.ml", line 33, characters 17-33",
"Called from file "src/pipe.ml", line 873, characters 10-13",
"Called from file "src/job_queue.ml" (inlined), line 131, characters 2-5",
"Called from file "src/job_queue.ml", line 171, characters 6-47",
"Caught by monitor coda"
]
],
"backtrace": [
"Raised at file "format.ml" (inlined), line 242, characters 35-52",
"Called from file "format.ml", line 469, characters 8-33",
"Called from file "format.ml", line 484, characters 6-24"
]
}
2021-11-03 23:15:34 UTC [Info] Coda daemon is booting up; built with commit "fe51f1ef5cfbcb4d5f559af39080963cf28ff1af" on branch "master"