Found a data race in GCWorker

[winkyao]

Bug Report

Original report: https://ask.pingcap.com/t/data-race-found-in-version/474

Application environment: 3 TiKV nodes, 3 PD nodes, 2 TiDB nodes in a local cluster
PoC environment
TiDB version: 6.6.0 (commit b417ad0)
Reproduction method: Brought the environment up and ran a simple bank account transaction workload, using the antithesis testing platform.

Problem: 3 Data Races observed:
First DATA RACE:

WARNING: DATA RACE
Read at 0x00c001e86958 by goroutine 9152:
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).resolveLocks()
      /tidb/store/gcworker/gc_worker.go:1169 +0xb2
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).runGCJob()
      /tidb/store/gcworker/gc_worker.go:767 +0x117
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).leaderTick.func1()
      /tidb/store/gcworker/gc_worker.go:444 +0x6c

Previous write at 0x00c001e86958 by goroutine 1007:
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).checkLeader()
      /tidb/store/gcworker/gc_worker.go:1920 +0x1cb
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).tick()
      /tidb/store/gcworker/gc_worker.go:286 +0x64
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).start()
      /tidb/store/gcworker/gc_worker.go:229 +0x659
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).Start.func1()
      /tidb/store/gcworker/gc_worker.go:120 +0x64

Goroutine 9152 (running) created at:
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).leaderTick()
      /tidb/store/gcworker/gc_worker.go:443 +0xb5a
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).tick()
      /tidb/store/gcworker/gc_worker.go:293 +0x8f
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).start()
      /tidb/store/gcworker/gc_worker.go:229 +0x659
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).Start.func1()
      /tidb/store/gcworker/gc_worker.go:120 +0x64

Goroutine 1007 (running) created at:
  github.com/pingcap/tidb/store/gcworker.(*GCWorker).Start()
      /tidb/store/gcworker/gc_worker.go:120 +0x1f7
  github.com/pingcap/tidb/store/driver.(*tikvStore).StartGCWorker()
      /tidb/store/driver/tikv_driver.go:320 +0x98
  github.com/pingcap/tidb/session.BootstrapSession()
      /tidb/session/session.go:3467 +0x1aaa
  github.com/pingcap/tidb/domain.(*Domain).rebuildSysVarCache()
      /tidb/domain/sysvar_cache.go:147 +0x8d0
  github.com/pingcap/tidb/sessionctx/variable.glob..func174()
      /tidb/sessionctx/variable/sysvar.go:904 +0x5e
  github.com/pingcap/tidb/sessionctx/variable.(*SysVar).ValidateWithRelaxedValidation()
      /tidb/sessionctx/variable/variable.go:361 +0x1c7
  github.com/pingcap/tidb/domain.(*Domain).rebuildSysVarCache()
      /tidb/domain/sysvar_cache.go:146 +0x844
  github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop()
      /tidb/domain/domain.go:1448 +0xa8
  github.com/pingcap/tidb/session.BootstrapSession()
      /tidb/session/session.go:3350 +0x6d3
  github.com/pingcap/tidb/domain.(*Domain).GetSessionCache()
      /tidb/domain/sysvar_cache.go:62 +0x59
  github.com/pingcap/tidb/session.(*session).loadCommonGlobalVariablesIfNeeded()
      /tidb/session/session.go:3669 +0x104
  github.com/pingcap/tidb/session.(*session).ExecuteStmt()
      /tidb/session/session.go:2148 +0x145
  github.com/pingcap/tidb/session.(*session).ExecuteInternal()
      /tidb/session/session.go:1678 +0x31b
  github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop()
      /tidb/domain/domain.go:1392 +0x130
  github.com/pingcap/tidb/session.BootstrapSession()
      /tidb/session/session.go:3343 +0x684
  main.createStoreAndDomain()
      /tidb/tidb-server/main.go:351 +0x304
  main.main()

Data race detected.

[davidsearle-antithesis]

This is Dave from Antithesis who originally logged this in the TiDB forum.

[zyguan]

@3pointer @joccau PTAL, there is data race on logBackupEnabled.

[davidsearle-antithesis]

I have two other data races I can share with you. Who can I talk to, to show how I'm finding these....?

…

On Tue, 21 Mar 2023 at 08:44, zyguan ***@***.***> wrote: @3pointer <https://github.com/3pointer> @joccau <https://github.com/joccau> PTAL, there is data race on logBackupEnabled. — Reply to this email directly, view it on GitHub <#42405 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT4N6ZYKG3TETINNBN2SZL3W5FS5TANCNFSM6AAAAAAWBEHGGU> . You are receiving this because you commented.Message ID: ***@***.***>

-- *-----------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity for whom they are addressed. If you received this message in error, please notify the sender and remove it from your system.*

[zyguan]

I have two other data races I can share with you. Who can I talk to, to show how I'm finding these....?

Thanks, @davidsearle-antithesis . Can you post the stacks of the other two data races here? In most cases, the stack info is enough to investigate data race issues.

[davidsearle-antithesis]

I have a system which can fuzz distributed architectures to find rare edge cases. This is being used to identify these data races. Who in Pingcap can I talk to about this? Pingcap could license this moving forward.

On Tue, 21 Mar 2023 at 10:11, zyguan ***@***.***> wrote: I have two other data races I can share with you. Who can I talk to, to show how I'm finding these....? Thanks, @davidsearle-antithesis <https://github.com/davidsearle-antithesis> . Can you post the stacks of the other two data races here? In most cases, the stack info is enough to investigate data race issues. — Reply to this email directly, view it on GitHub <#42405 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT4N6Z4MN22BJPUP5CGIOJDW5F5GPANCNFSM6AAAAAAWBEHGGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- David Searle | EMEA Field Operations | Antithesis +44 (7919) 352561 | ***@***.***

…

-- *-----------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity for whom they are addressed. If you received this message in error, please notify the sender and remove it from your system.*

[mjonss]

Just as a note, I and Daniël had a meeting with David yesterday and will follow up from there regarding possible collaborations.

[BornChanger]

@davidsearle-antithesis can you please tell me if the problem is still there against current TiDB version?

[3pointer]

should be fixed in #40759, the race code are removed in this PR

[cfzjywxk]

@MyonKeminta
Could you help to verify if it still exists?

[MyonKeminta]

@3pointer Did you encounter this issue again? Is there any more details?

[davidsearle-antithesis]

There are plenty more details however we are a licensed Saas provider. If you would like a meeting to discuss more then please let me know. Check www.antithesis.com for more information David Searle EMEA Field Operations +44 (7919) 352-561 [image: antithesis_logo_email.png] <https://antithesis.com/?utm_source=email&utm_medium=signature>

…

On Mon, 6 May 2024 at 08:32, MyonKeminta ***@***.***> wrote: @3pointer <https://github.com/3pointer> Did you encounter this issue again? Is there any more details? — Reply to this email directly, view it on GitHub <#42405 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT4N6Z6A4AO5YXZ24VIF2P3ZA4WYNAVCNFSM6AAAAAAWBEHGGWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJVGM2TQOJSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *-----------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity for whom they are addressed. If you received this message in error, please notify the sender and remove it from your system.*