Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix a race in tipset cache usage #4282

Merged
merged 1 commit into from
Oct 10, 2020
Merged

fix a race in tipset cache usage #4282

merged 1 commit into from
Oct 10, 2020

Conversation

Stebalien
Copy link
Member

This tipset cache is shared between multiple services and is called from multiple places.

@Stebalien
Copy link
Member Author

logs 
==================

WARNING: DATA RACE

Write at 0x00c001963458 by goroutine 877:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).add()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:59 +0x359

github.com/filecoin-project/lotus/chain/events.(*heightEvents).headChangeAt()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_height.go:84 +0x724

github.com/filecoin-project/lotus/chain/events.(*Events).headChange()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:175 +0x20a

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChangesOnce()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:154 +0x7a4

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChanges()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:94 +0x11e


Previous read at 0x00c001963458 by goroutine 1042:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).best()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:127 +0x64

github.com/filecoin-project/lotus/chain/events.(*hcEvents).onHeadChanged()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_called.go:310 +0xe4

github.com/filecoin-project/lotus/chain/events.(*watcherEvents).StateChanged()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_called.go:443 +0x137

github.com/filecoin-project/lotus/markets/storageadapter.(*ProviderNodeAdapter).OnDealExpiredOrSlashed()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/provider.go:475 +0x9fb

github.com/filecoin-project/go-fil-markets/storagemarket/impl/providerstates.WaitForDealCompletion()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/storagemarket/impl/providerstates/provider_states.go:454 +0x201

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine/fsm.fsmHandler.handler.func1()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/fsm/fsm.go:170 +0x484

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine.(*StateMachine).run.func3()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/machine.go:102 +0x27c


Goroutine 877 (running) created at:

github.com/filecoin-project/lotus/chain/events.NewEvents()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:83 +0x7c4

github.com/filecoin-project/lotus/markets/storageadapter.NewProviderNodeAdapter.func1()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/provider.go:64 +0xc8

runtime.call64()

/usr/lib/go/src/runtime/asm_amd64.s:541 +0x3d

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/lotus/node.as.func2()

/home/steb/src/github.com/filecoin-project/lotus/node/options.go:140 +0xb5

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

go.uber.org/dig.defaultInvoker()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:284 +0x7a

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:710 +0x2d0

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7

go.uber.org/dig.(*Container).Invoke()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:432 +0x372

go.uber.org/fx.(*App).executeInvokes()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/app.go:547 +0x493

go.uber.org/fx.New()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/app.go:345 +0x83b

github.com/filecoin-project/lotus/node.New()

/home/steb/src/github.com/filecoin-project/lotus/node/builder.go:570 +0x64e

github.com/filecoin-project/lotus/node/test.CreateTestStorageNode()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:107 +0x11b3

github.com/filecoin-project/lotus/node/test.mockSbBuilderOpts()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:445 +0x24ad

github.com/filecoin-project/lotus/node/test.MockSbBuilder()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:149 +0xa8

github.com/filecoin-project/lotus/api/test.TestDealFlow()

/home/steb/src/github.com/filecoin-project/lotus/api/test/deals.go:51 +0x148

github.com/filecoin-project/lotus/node_test.TestAPIDealFlow.func1()

/home/steb/src/github.com/filecoin-project/lotus/node/node_test.go:42 +0x54

testing.tRunner()

/usr/lib/go/src/testing/testing.go:1127 +0x202


Goroutine 1042 (finished) created at:

github.com/filecoin-project/go-statemachine.(*StateMachine).run()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/machine.go:100 +0x7ab


==================

WARNING: DATA RACE

Write at 0x00c00055ff58 by goroutine 397:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).add()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:59 +0x359

github.com/filecoin-project/lotus/chain/events.(*heightEvents).headChangeAt()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_height.go:84 +0x724

github.com/filecoin-project/lotus/chain/events.(*Events).headChange()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:175 +0x20a

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChangesOnce()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:154 +0x7a4

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChanges()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:94 +0x11e


Previous read at 0x00c00055ff58 by goroutine 250:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).best()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:127 +0x64

github.com/filecoin-project/lotus/chain/events.(*hcEvents).onHeadChanged()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_called.go:310 +0xe4

github.com/filecoin-project/lotus/chain/events.(*watcherEvents).StateChanged()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_called.go:443 +0x137

github.com/filecoin-project/lotus/markets/storageadapter.(*ClientNodeAdapter).OnDealExpiredOrSlashed()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/client.go:406 +0x9e4

github.com/filecoin-project/go-fil-markets/storagemarket/impl/clientstates.WaitForDealCompletion()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/storagemarket/impl/clientstates/client_states.go:270 +0x193

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine/fsm.fsmHandler.handler.func1()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/fsm/fsm.go:170 +0x484

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine.(*StateMachine).run.func3()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/machine.go:102 +0x27c


Goroutine 397 (running) created at:

github.com/filecoin-project/lotus/chain/events.NewEvents()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:83 +0x7c4

github.com/filecoin-project/lotus/markets/storageadapter.NewClientNodeAdapter()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/client.go:61 +0x144

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/lotus/node.as.func2()

/home/steb/src/github.com/filecoin-project/lotus/node/options.go:140 +0xb5

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

go.uber.org/dig.defaultInvoker()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:284 +0x7a

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:710 +0x2d0

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramObjectField.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:384 +0x1c3

go.uber.org/dig.paramObject.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:311 +0x160

go.uber.org/dig.(*paramObject).Build()

:1 +0xe7

go.uber.org/dig.paramObjectField.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:384 +0x1c3

go.uber.org/dig.paramObject.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:311 +0x160

go.uber.org/dig.(*paramObject).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7

go.uber.org/dig.(*Container).Invoke()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:432 +0x372

go.uber.org/fx.(*App).executeInvokes()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/app.go:547 +0x493

go.uber.org/fx.New()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/app.go:345 +0x83b

github.com/filecoin-project/lotus/node.New()

/home/steb/src/github.com/filecoin-project/lotus/node/builder.go:570 +0x64e

github.com/filecoin-project/lotus/node/test.mockSbBuilderOpts()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:396 +0x184e

github.com/filecoin-project/lotus/node/test.MockSbBuilder()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:149 +0xa8

github.com/filecoin-project/lotus/api/test.TestDealFlow()

/home/steb/src/github.com/filecoin-project/lotus/api/test/deals.go:51 +0x148

github.com/filecoin-project/lotus/node_test.TestAPIDealFlow.func1()

/home/steb/src/github.com/filecoin-project/lotus/node/node_test.go:42 +0x54

testing.tRunner()

/usr/lib/go/src/testing/testing.go:1127 +0x202


Goroutine 250 (finished) created at:

github.com/filecoin-project/go-statemachine.(*StateMachine).run()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/machine.go:100 +0x7ab


==================

WARNING: DATA RACE

Write at 0x00c001f83118 by goroutine 898:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).add()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:59 +0x359

github.com/filecoin-project/lotus/chain/events.(*heightEvents).headChangeAt()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_height.go:84 +0x724

github.com/filecoin-project/lotus/chain/events.(*Events).headChange()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:175 +0x20a

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChangesOnce()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:154 +0x7a4

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChanges()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:94 +0x11e


Previous read at 0x00c001f83118 by goroutine 706:

??()

-:0 +0xffffffffffffffff

github.com/ipfs/go-ipfs-ds-help.NewKeyFromBinary()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/[email protected]/key.go:16 +0x16c

github.com/ipfs/go-ipfs-ds-help.MultihashToDsKey()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/[email protected]/key.go:30 +0x8d

github.com/ipfs/go-ipfs-blockstore.(*blockstore).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/[email protected]/blockstore.go:124 +0x56

github.com/ipfs/go-ipfs-blockstore.(*idstore).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/[email protected]/idstore.go:62 +0x136

github.com/ipfs/go-ipfs-blockstore.(*arccache).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/[email protected]/arc_cache.go:118 +0xd8

github.com/ipfs/go-ipfs-blockstore.(*bloomcache).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/[email protected]/bloom_cache.go:159 +0xa6

github.com/ipfs/go-ipfs-blockstore.(*idstore).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/[email protected]/idstore.go:62 +0x136

github.com/ipfs/go-ipld-cbor.(*BasicIpldStore).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/[email protected]/store.go:44 +0x8a

github.com/filecoin-project/specs-actors/actors/util/adt.(*wstore).Get()

:1 +0xb6

github.com/filecoin-project/go-amt-ipld/v2.LoadAMT()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-amt-ipld/[email protected]/amt.go:56 +0x101

github.com/filecoin-project/specs-actors/actors/util/adt.AsArray()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/actors/util/adt/array.go:22 +0xe4

github.com/filecoin-project/lotus/chain/actors/builtin/market.(*state0).States()

/home/steb/src/github.com/filecoin-project/lotus/chain/actors/builtin/market/v0.go:61 +0xc7

github.com/filecoin-project/lotus/chain/stmgr.GetStorageDeal()

/home/steb/src/github.com/filecoin-project/lotus/chain/stmgr/utils.go:325 +0x566

github.com/filecoin-project/lotus/node/impl/full.(*StateAPI).StateMarketStorageDeal()

/home/steb/src/github.com/filecoin-project/lotus/node/impl/full/state.go:658 +0x237

github.com/filecoin-project/lotus/node/impl.(*FullNodeAPI).StateMarketStorageDeal()

:1 +0x8f

github.com/filecoin-project/lotus/api/test.(*TestNode).StateMarketStorageDeal()

:1 +0xaa

github.com/filecoin-project/lotus/markets/storageadapter.(*ProviderNodeAdapter).OnDealExpiredOrSlashed()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/provider.go:393 +0x28d

github.com/filecoin-project/go-fil-markets/storagemarket/impl/providerstates.WaitForDealCompletion()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/storagemarket/impl/providerstates/provider_states.go:454 +0x201

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine/fsm.fsmHandler.handler.func1()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/fsm/fsm.go:170 +0x484

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine.(*StateMachine).run.func3()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/machine.go:102 +0x27c


Goroutine 898 (running) created at:

github.com/filecoin-project/lotus/chain/events.NewEvents()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:83 +0x7c4

github.com/filecoin-project/lotus/markets/storageadapter.NewProviderNodeAdapter.func1()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/provider.go:64 +0xc8

runtime.call64()

/usr/lib/go/src/runtime/asm_amd64.s:541 +0x3d

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/lotus/node.as.func2()

/home/steb/src/github.com/filecoin-project/lotus/node/options.go:140 +0xb5

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

go.uber.org/dig.defaultInvoker()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:284 +0x7a

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:710 +0x2d0

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7

go.uber.org/dig.(*Container).Invoke()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:432 +0x372

go.uber.org/fx.(*App).executeInvokes()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/app.go:547 +0x493

go.uber.org/fx.New()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/app.go:345 +0x83b

github.com/filecoin-project/lotus/node.New()

/home/steb/src/github.com/filecoin-project/lotus/node/builder.go:570 +0x64e

github.com/filecoin-project/lotus/node/test.CreateTestStorageNode()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:107 +0x11b3

github.com/filecoin-project/lotus/node/test.mockBuilderOpts()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:281 +0x2504

github.com/filecoin-project/lotus/node/test.Builder()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:145 +0xa8

github.com/filecoin-project/lotus/api/test.TestDealFlow()

/home/steb/src/github.com/filecoin-project/lotus/api/test/deals.go:51 +0x148

github.com/filecoin-project/lotus/node_test.TestAPIDealFlowReal.func3()

/home/steb/src/github.com/filecoin-project/lotus/node/node_test.go:78 +0x54

testing.tRunner()

/usr/lib/go/src/testing/testing.go:1127 +0x202


Goroutine 706 (finished) created at:

github.com/filecoin-project/go-statemachine.(*StateMachine).run()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/machine.go:100 +0x7ab


==================

WARNING: DATA RACE

Write at 0x00c000cd95d8 by goroutine 809:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).add()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:59 +0x359

github.com/filecoin-project/lotus/chain/events.(*heightEvents).headChangeAt()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_height.go:84 +0x724

github.com/filecoin-project/lotus/chain/events.(*Events).headChange()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:175 +0x20a

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChangesOnce()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:154 +0x7a4

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChanges()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:94 +0x11e


Previous read at 0x00c000cd95d8 by goroutine 347:

??()

-:0 +0xffffffffffffffff

github.com/filecoin-project/go-amt-ipld/v2.(*Root).UnmarshalCBOR()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-amt-ipld/[email protected]/cbor_gen.go:98 +0x354

github.com/ipfs/go-ipld-cbor.(*BasicIpldStore).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/[email protected]/store.go:51 +0x204

github.com/filecoin-project/specs-actors/actors/util/adt.(*wstore).Get()

:1 +0xb6

github.com/filecoin-project/go-amt-ipld/v2.LoadAMT()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-amt-ipld/[email protected]/amt.go:56 +0x101

github.com/filecoin-project/specs-actors/actors/util/adt.AsArray()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/actors/util/adt/array.go:22 +0xe4

github.com/filecoin-project/lotus/chain/actors/builtin/market.(*state0).States()

/home/steb/src/github.com/filecoin-project/lotus/chain/actors/builtin/market/v0.go:61 +0xc7

github.com/filecoin-project/lotus/chain/stmgr.GetStorageDeal()

/home/steb/src/github.com/filecoin-project/lotus/chain/stmgr/utils.go:325 +0x566

github.com/filecoin-project/lotus/node/impl/full.(*StateAPI).StateMarketStorageDeal()

/home/steb/src/github.com/filecoin-project/lotus/node/impl/full/state.go:658 +0x237

github.com/filecoin-project/lotus/markets/storageadapter.(*ClientNodeAdapter).OnDealExpiredOrSlashed()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/client.go:324 +0x257

github.com/filecoin-project/go-fil-markets/storagemarket/impl/clientstates.WaitForDealCompletion()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/storagemarket/impl/clientstates/client_states.go:270 +0x193

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine/fsm.fsmHandler.handler.func1()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/fsm/fsm.go:170 +0x484

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine.(*StateMachine).run.func3()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/machine.go:102 +0x27c


Goroutine 809 (running) created at:

github.com/filecoin-project/lotus/chain/events.NewEvents()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:83 +0x7c4

github.com/filecoin-project/lotus/markets/storageadapter.NewClientNodeAdapter()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/client.go:61 +0x144

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/lotus/node.as.func2()

/home/steb/src/github.com/filecoin-project/lotus/node/options.go:140 +0xb5

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

go.uber.org/dig.defaultInvoker()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:284 +0x7a

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:710 +0x2d0

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramObjectField.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:384 +0x1c3

go.uber.org/dig.paramObject.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:311 +0x160

go.uber.org/dig.(*paramObject).Build()

:1 +0xe7

go.uber.org/dig.paramObjectField.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:384 +0x1c3

go.uber.org/dig.paramObject.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:311 +0x160

go.uber.org/dig.(*paramObject).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7

go.uber.org/dig.(*Container).Invoke()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/dig.go:432 +0x372

go.uber.org/fx.(*App).executeInvokes()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/app.go:547 +0x493

go.uber.org/fx.New()

/home/steb/.local/share/go/pkg/mod/go.uber.org/[email protected]/app.go:345 +0x83b

github.com/filecoin-project/lotus/node.New()

/home/steb/src/github.com/filecoin-project/lotus/node/builder.go:570 +0x64e

github.com/filecoin-project/lotus/node/test.mockBuilderOpts()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:239 +0x18f2

github.com/filecoin-project/lotus/node/test.Builder()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:145 +0xa8

github.com/filecoin-project/lotus/api/test.TestDealFlow()

/home/steb/src/github.com/filecoin-project/lotus/api/test/deals.go:51 +0x148

github.com/filecoin-project/lotus/node_test.TestAPIDealFlowReal.func3()

/home/steb/src/github.com/filecoin-project/lotus/node/node_test.go:78 +0x54

testing.tRunner()

/usr/lib/go/src/testing/testing.go:1127 +0x202


Goroutine 347 (finished) created at:

github.com/filecoin-project/go-statemachine.(*StateMachine).run()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/[email protected]/machine.go:100 +0x7ab

@Stebalien Stebalien force-pushed the steb/fix-tipset-cache-race branch from 2860055 to 24dd6da Compare October 10, 2020 00:26
Stebalien
Stebalien commented Oct 10, 2020
View reviewed changes
chain/events/tscache.go
@@ -95,14 +104,18 @@ func (tsc *tipSetCache) getNonNull(height abi.ChainEpoch) (*types.TipSet, error)
}

func (tsc *tipSetCache) get(height abi.ChainEpoch) (*types.TipSet, error) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit nasty but I'm trying to avoid calling into the storage subsystem while holding a lock.

@Stebalien Stebalien force-pushed the steb/fix-tipset-cache-race branch from 24dd6da to f925f0f Compare October 10, 2020 00:31
@Stebalien
fix a race in tipset cache usage
f32b3bf 
This tipset cache is shared between multiple services and is called from
multiple places.
@Stebalien Stebalien force-pushed the steb/fix-tipset-cache-race branch from f925f0f to f32b3bf Compare October 10, 2020 00:32
@magik6k magik6k merged commit 9a26896 into master Oct 10, 2020
@magik6k magik6k deleted the steb/fix-tipset-cache-race branch October 10, 2020 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@magik6k magik6k magik6k approved these changes

@Kubuxu Kubuxu Awaiting requested review from Kubuxu

@whyrusleeping whyrusleeping Awaiting requested review from whyrusleeping

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Stebalien @magik6k