Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Removed/Terminated sectors still get Event Logs (state machine error: %!s(<nil>)) #9968

Closed
10 of 18 tasks
Tracked by #10338
lvschouwen opened this issue Jan 5, 2023 · 0 comments · Fixed by #10363
Closed
10 of 18 tasks
Tracked by #10338

Removed/Terminated sectors still get Event Logs (state machine error: %!s(<nil>)) #9968

lvschouwen opened this issue Jan 5, 2023 · 0 comments · Fixed by #10363
Labels
area/sealing kind/bug Kind: Bug need/analysis Hint: Needs Analysis
Milestone
Lotus-Miner Backl...

Comments

@lvschouwen
Copy link

lvschouwen commented Jan 5, 2023 •
edited by Reiers
Loading

Checklist

  • This is not a security-related bug/issue. If it is, please follow please follow the security policy.
  • This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
  • This is not a new feature request. If it is, please file a feature request instead.
  • This is not an enhancement request. If it is, please file a improvement suggestion instead.
  • I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
  • I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
  • I did not make any code changes to lotus.

Lotus component

  • lotus daemon - chain sync
  • lotus miner - mining and block production
  • lotus miner/worker - sealing
  • lotus miner - proving(WindowPoSt)
  • lotus miner/market - storage deal
  • lotus miner/market - retrieval deal
  • lotus miner/market - data transfer
  • lotus client
  • lotus JSON-RPC API
  • lotus message management (mpool)
  • Other

Lotus Version

Daemon:  1.19.0+mainnet+git.64059ca87+api1.5.0
Local: lotus version 1.19.0+mainnet+git.64059ca87

Daemon:  1.19.0+mainnet+git.64059ca87+api1.5.0
Local: lotus-miner version 1.19.0+mainnet+git.64059ca87

Describe the Bug

It seems that sectors with status Removed or Terminated still receive regular logging if you request the status of that sector via "lotus-miner sectors status -l"

root@R31-Core01:~# lotus-miner sectors status -l 0
SectorID:	0
Status:		Removed
CIDcommD:	baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
CIDcommR:	bagboea4b5abcbz5vbl3x6c2qrgmzrenywvahuz6ewclqxg2xz5cv2ggmm7ag22rd
Ticket:		b00450207b5fe884d3bc9e0b2e03ee6a6ae616f62e1f89561615c1b1aaf89f21
TicketH:	2245272
Seed:		b4e912f6cbeca234131042cc4d9774c7b02146853038edd9843f5803572b258b
SeedH:		2246950
Precommit:	bafy2bzaceaymqxj73dts7lzzymp6one2wc75vj6qw6ist4g4kussoipmvpdmw
Commit:		<nil>
Deals:		[0]
Retries:	0
--------
Event Log:
...
90.	2022-11-16 16:10:03 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
91.	2022-12-06 16:35:22 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
92.	2022-12-27 12:33:44 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
93.	2022-12-27 13:58:07 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
94.	2022-12-30 10:18:49 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
95.	2023-01-04 09:32:19 +0100 CET:[error;<nil>]	state machine error: %!s(<nil>)

Logging Information

lotus-miner sectors status -l 0
SectorID:	0
Status:		Removed
CIDcommD:	baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
CIDcommR:	bagboea4b5abcbz5vbl3x6c2qrgmzrenywvahuz6ewclqxg2xz5cv2ggmm7ag22rd
Ticket:		b00450207b5fe884d3bc9e0b2e03ee6a6ae616f62e1f89561615c1b1aaf89f21
TicketH:	2245272
Seed:		b4e912f6cbeca234131042cc4d9774c7b02146853038edd9843f5803572b258b
SeedH:		2246950
Precommit:	bafy2bzaceaymqxj73dts7lzzymp6one2wc75vj6qw6ist4g4kussoipmvpdmw
Commit:		<nil>
Deals:		[0]
Retries:	0
--------
Event Log:
0.	2022-10-13 16:30:23 +0200 CEST:	[event;sealing.SectorStartCC]	{"User":{"ID":0,"SectorType":8}}
1.	2022-10-13 22:05:59 +0200 CEST:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}]}}
2.	2022-10-13 22:05:59 +0200 CEST:	[event;sealing.SectorTicket]	{"User":{"TicketValue":"sARQIHtf6ITTvJ4LLgPuamrmFvYuH4lWFhXBsar4nyE=","TicketEpoch":2245272}}
3.	2022-10-14 01:05:44 +0200 CEST:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJzQVJRSUh0ZjZJVFR2SjRMTGdQdWFtcm1Gdll1SDRsV0ZoWEJzYXI0bnlFPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvRGF0YVBvb2xfSkJPRDAxXzAyL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjIxNDc0ODM2NDd9LCJsYWJlbHMiOnsiU3RhY2tlZERyZzMyR2lCVjEiOnsiX2giOm51bGwsImxhYmVscyI6W3siaWQiOiJsYXllci0xIiwicGF0aCI6Ii9EYXRhUG9vbF9KQk9EMDFfMDIvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTIiLCJwYXRoIjoiL0RhdGFQb29sX0pCT0QwMV8wMi9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvRGF0YVBvb2xfSkJPRDAxXzAyL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci00IiwicGF0aCI6Ii9EYXRhUG9vbF9KQk9EMDFfMDIvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTUiLCJwYXRoIjoiL0RhdGFQb29sX0pCT0QwMV8wMi9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvRGF0YVBvb2xfSkJPRDAxXzAyL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci03IiwicGF0aCI6Ii9EYXRhUG9vbF9KQk9EMDFfMDIvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTgiLCJwYXRoIjoiL0RhdGFQb29sX0pCT0QwMV8wMi9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvRGF0YVBvb2xfSkJPRDAxXzAyL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci0xMCIsInBhdGgiOiIvRGF0YVBvb2xfSkJPRDAxXzAyL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci0xMSIsInBhdGgiOiIvRGF0YVBvb2xfSkJPRDAxXzAyL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
4.	2022-10-14 02:06:42 +0200 CEST:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcbz5vbl3x6c2qrgmzrenywvahuz6ewclqxg2xz5cv2ggmm7ag22rd"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
5.	2022-10-14 02:06:42 +0200 CEST:	[event;sealing.SectorPreCommitBatch]	{"User":{}}
6.	2022-10-14 03:19:41 +0200 CEST:	[event;sealing.SectorPreCommitBatchSent]	{"User":{"Message":{"/":"bafy2bzaceaymqxj73dts7lzzymp6one2wc75vj6qw6ist4g4kussoipmvpdmw"}}}
7.	2022-10-14 03:23:00 +0200 CEST:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":[{"/":"bafy2bzaceckxm5yn47uzjrhmkqjhoxojr474dsgk3njifdmdw5wbbd7d7zrik"},{"/":"bafy2bzacedp2enb3zs6aed2xfblug66cikxn7hwrbf4acp4oc6oaszouimfnm"}]}}
8.	2022-10-14 04:37:59 +0200 CEST:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"tOkS9svsojQTEELMTZd0x7AhRoUwOO3ZhD9YA1crJYs=","SeedEpoch":2246950}}
9.	2022-10-14 04:42:10 +0200 CEST:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 0: Rust panic: no unwind information [name: R41-PC2Worker01-A]: Rust panic: no unwind information
10.	2022-10-14 04:43:10 +0200 CEST:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
11.	2022-10-14 04:46:24 +0200 CEST:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 0: Rust panic: no unwind information [name: R41-PC2Worker01-A]: Rust panic: no unwind information
12.	2022-10-14 04:47:24 +0200 CEST:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
13.	2022-10-14 04:50:08 +0200 CEST:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 0: Rust panic: no unwind information [name: R41-PC2Worker01-A]: Rust panic: no unwind information
14.	2022-10-14 04:51:08 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	consecutive compute fails
15.	2022-10-14 04:52:08 +0200 CEST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
16.	2022-10-14 07:37:02 +0200 CEST:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJzQVJRSUh0ZjZJVFR2SjRMTGdQdWFtcm1Gdll1SDRsV0ZoWEJzYXI0bnlFPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvczEvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MjE0NzQ4MzY0N30sImxhYmVscyI6eyJTdGFja2VkRHJnMzJHaUJWMSI6eyJfaCI6bnVsbCwibGFiZWxzIjpbeyJpZCI6ImxheWVyLTEiLCJwYXRoIjoiL3MxL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci0yIiwicGF0aCI6Ii9zMS9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvczEvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTQiLCJwYXRoIjoiL3MxL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci01IiwicGF0aCI6Ii9zMS9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvczEvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTciLCJwYXRoIjoiL3MxL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci04IiwicGF0aCI6Ii9zMS9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvczEvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTEwIiwicGF0aCI6Ii9zMS9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTEiLCJwYXRoIjoiL3MxL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
17.	2022-10-14 07:55:41 +0200 CEST:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcbz5vbl3x6c2qrgmzrenywvahuz6ewclqxg2xz5cv2ggmm7ag22rd"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
18.	2022-10-14 07:55:41 +0200 CEST:	[event;sealing.SectorPreCommitBatch]	{"User":{}}
19.	2022-10-14 07:55:41 +0200 CEST:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":[{"/":"bafy2bzaceczzlaphp74livr2hrbagico5cql4khwri7c76hya7drarusukdqi"},{"/":"bafy2bzacebfi6e3pqnocxsclkcxy3v5eyyb6lnsbdj53lxhgmkjwxdxcrfyjk"},{"/":"bafy2bzacebuccgtobomfi5zmuawt7j7vhvviaiby5kml4ttuda7oec6kyctrs"},{"/":"bafy2bzacecatrf35wxvpufyrcw6jgsho6u6fcnjqthhrjqofnslxd33rmwrls"}]}}
20.	2022-10-14 07:55:41 +0200 CEST:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"tOkS9svsojQTEELMTZd0x7AhRoUwOO3ZhD9YA1crJYs=","SeedEpoch":2246950}}
21.	2022-10-14 07:59:35 +0200 CEST:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 0: Rust panic: no unwind information [name: R41-PC2Worker01-A]: Rust panic: no unwind information
22.	2022-10-14 08:00:35 +0200 CEST:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
23.	2022-10-14 08:03:24 +0200 CEST:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 0: Rust panic: no unwind information [name: R41-PC2Worker01-A]: Rust panic: no unwind information
24.	2022-10-14 08:04:24 +0200 CEST:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
25.	2022-10-14 08:07:09 +0200 CEST:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 0: Rust panic: no unwind information [name: R41-PC2Worker01-A]: Rust panic: no unwind information
26.	2022-10-14 08:08:09 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	consecutive compute fails
27.	2022-10-14 08:09:09 +0200 CEST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
28.	2022-10-14 10:33:50 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 101: worker [name: R41-PC1Node1] restarted
29.	2022-10-14 10:34:50 +0200 CEST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
30.	2022-10-14 13:16:21 +0200 CEST:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJzQVJRSUh0ZjZJVFR2SjRMTGdQdWFtcm1Gdll1SDRsV0ZoWEJzYXI0bnlFPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvczIvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MjE0NzQ4MzY0N30sImxhYmVscyI6eyJTdGFja2VkRHJnMzJHaUJWMSI6eyJfaCI6bnVsbCwibGFiZWxzIjpbeyJpZCI6ImxheWVyLTEiLCJwYXRoIjoiL3MyL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci0yIiwicGF0aCI6Ii9zMi9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvczIvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTQiLCJwYXRoIjoiL3MyL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci01IiwicGF0aCI6Ii9zMi9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvczIvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTciLCJwYXRoIjoiL3MyL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci04IiwicGF0aCI6Ii9zMi9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvczIvY2FjaGUvcy10MDE5Mzc2NDItMCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTEwIiwicGF0aCI6Ii9zMi9jYWNoZS9zLXQwMTkzNzY0Mi0wIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTEiLCJwYXRoIjoiL3MyL2NhY2hlL3MtdDAxOTM3NjQyLTAiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
31.	2022-10-14 13:24:33 +0200 CEST:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcbz5vbl3x6c2qrgmzrenywvahuz6ewclqxg2xz5cv2ggmm7ag22rd"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
32.	2022-10-14 13:24:33 +0200 CEST:	[event;sealing.SectorPreCommitBatch]	{"User":{}}
33.	2022-10-14 13:24:33 +0200 CEST:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":[{"/":"bafy2bzacebzhsl7so6zbxp6pmemqbiugbiot4ct3oqywq4lil2luooehvobp4"},{"/":"bafy2bzacedgnnyw4mghrv36frfi34g6kx2h5dpy6homxj5yzrg7huedvrhbog"},{"/":"bafy2bzacea6gyvqikqm3xct2mubgyr73bjf4klnaiwm53zpgazwsjg6umxpyi"},{"/":"bafy2bzacecrvmbeqrher56vyeqmjriupl7yunjcbtc45bxahe63zdisq6dzgc"},{"/":"bafy2bzacedjgsg43hobdoewcz3ez6ivi77hmv36tbgtbteh5wnwvmi3gzeq64"},{"/":"bafy2bzaceby3axi54jml77id2oumhah4fgh7v6thxg25wi7cw6s2s2a5lxd3c"},{"/":"bafy2bzacedclswjarcjoyyh3f3y33wfbqmeukx3c2txxzez3bl3ssf27lfemo"}]}}
34.	2022-10-14 13:24:33 +0200 CEST:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"tOkS9svsojQTEELMTZd0x7AhRoUwOO3ZhD9YA1crJYs=","SeedEpoch":2246950}}
35.	2022-10-14 13:26:44 +0200 CEST:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 0: Rust panic: no unwind information [name: R41-PC2Worker01-B]: Rust panic: no unwind information
36.	2022-10-14 13:27:44 +0200 CEST:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
37.	2022-10-14 13:29:34 +0200 CEST:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 0: Rust panic: no unwind information [name: R41-PC2Worker01-B]: Rust panic: no unwind information
38.	2022-10-14 13:30:34 +0200 CEST:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
39.	2022-10-14 13:32:26 +0200 CEST:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 0: Rust panic: no unwind information [name: R41-PC2Worker01-A]: Rust panic: no unwind information
40.	2022-10-14 13:33:26 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	consecutive compute fails
41.	2022-10-14 13:34:26 +0200 CEST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
42.	2022-10-14 15:20:47 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: task aborted
43.	2022-10-14 15:21:47 +0200 CEST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
44.	2022-10-14 15:36:54 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: task aborted
45.	2022-10-14 15:37:54 +0200 CEST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
46.	2022-10-14 15:38:54 +0200 CEST:	[event;sealing.SectorTerminate]	{"User":{}}
47.	2022-10-14 15:38:54 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: task aborted
48.	2022-10-14 15:38:54 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: task aborted
49.	2022-10-14 15:38:54 +0200 CEST:	[event;sealing.SectorTerminateFailed]	{"User":{}}
	sector was precommitted but not proven, remove instead of terminating
50.	2022-10-14 15:38:54 +0200 CEST:	[error;*xerrors.wrapError]	state machine error: running planner for state Terminating failed: planner for state Terminating received unexpected event sealing.SectorSealPreCommit1Failed ({User:seal pre commit(1) failed: storage call error 0: task aborted})
51.	2022-10-14 15:38:54 +0200 CEST:	[event;sealing.SectorTerminateFailed]	{"User":{}}
	sector was precommitted but not proven, remove instead of terminating
52.	2022-10-14 15:41:22 +0200 CEST:	[event;sealing.SectorRemove]	{"User":{}}
53.	2022-10-14 15:41:22 +0200 CEST:	[event;sealing.SectorRemoved]	{"User":{}}
54.	2022-10-14 15:41:22 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
55.	2022-10-14 18:30:44 +0200 CEST:	[event;sealing.SectorRemove]	{"User":{}}
56.	2022-10-14 18:30:44 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
57.	2022-10-14 20:24:01 +0200 CEST:	[event;sealing.SectorRemove]	{"User":{}}
58.	2022-10-14 20:24:01 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
59.	2022-10-14 23:12:44 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
60.	2022-10-15 08:49:10 +0200 CEST:	[event;sealing.SectorRemove]	{"User":{}}
61.	2022-10-15 08:49:10 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
62.	2022-10-15 08:49:17 +0200 CEST:	[event;sealing.SectorRemove]	{"User":{}}
63.	2022-10-15 08:49:17 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
64.	2022-10-15 08:49:18 +0200 CEST:	[event;sealing.SectorRemove]	{"User":{}}
65.	2022-10-15 08:49:18 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
66.	2022-10-15 16:05:40 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
67.	2022-10-15 16:27:09 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
68.	2022-10-15 16:56:32 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
69.	2022-10-15 17:12:47 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
70.	2022-10-15 17:40:35 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
71.	2022-10-16 11:42:06 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
72.	2022-10-17 23:05:05 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
73.	2022-10-19 10:33:08 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
74.	2022-10-20 22:30:44 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
75.	2022-10-20 22:50:53 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
76.	2022-10-21 08:19:14 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
77.	2022-10-21 09:09:20 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
78.	2022-10-21 11:12:48 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
79.	2022-10-21 11:22:16 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
80.	2022-10-21 11:28:55 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
81.	2022-10-22 08:28:37 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
82.	2022-10-26 17:49:44 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
83.	2022-10-27 11:35:41 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
84.	2022-10-27 12:39:02 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
85.	2022-10-27 12:43:20 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
86.	2022-10-27 12:52:03 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
87.	2022-10-28 18:29:41 +0200 CEST:	[error;<nil>]	state machine error: %!s(<nil>)
88.	2022-11-01 11:56:57 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
89.	2022-11-16 16:06:25 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
90.	2022-11-16 16:10:03 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
91.	2022-12-06 16:35:22 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
92.	2022-12-27 12:33:44 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
93.	2022-12-27 13:58:07 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
94.	2022-12-30 10:18:49 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)
95.	2023-01-04 09:32:19 +0100 CET:	[error;<nil>]	state machine error: %!s(<nil>)

Repo Steps

I've check this on multiple miners and I saw this on all of them.
@Reiers Reiers added need/analysis Hint: Needs Analysis area/sealing and removed need/triage labels Jan 5, 2023
@rjan90 rjan90 mentioned this issue Feb 24, 2023
Closed
@magik6k magik6k mentioned this issue Feb 28, 2023
Merged
7 tasks
@rjan90 rjan90 added this to the v1.21.0 milestone Feb 28, 2023
@rjan90 rjan90 moved this to 🧪In Testing in Lotus-Miner-V2 Mar 2, 2023
@rjan90 rjan90 moved this from 🧪In Testing to 👀 In Review in Lotus-Miner-V2 Mar 8, 2023
@rjan90 rjan90 moved this from 👀 In Review to ✅ Done - v1.21.0 in Lotus-Miner-V2 Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/sealing kind/bug Kind: Bug need/analysis Hint: Needs Analysis
Projects
Lotus-Miner-V2
Status: No status
Milestone
Lotus-Miner Backlog Sprint
Development

Successfully merging a pull request may close this issue.

fix: fsm: shutdown removed sectors FSMs filecoin-project/lotus
4 participants
@lvschouwen @rjan90 @jennijuju @Reiers