Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sessionctx/stmtctx: avoid unlock of unlocked mutex panic on Statement… #58629

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

chibiegg
Copy link

@chibiegg chibiegg commented Dec 31, 2024

What problem does this PR solve?

Issue Number: close #58600

Problem Summary:

What changed and how does it work?

Similar to #39368 .

In infoschema_reader.go , I modified the code so that RefCountOfStmtCtx is referenced in places where StatementContext was being used.

Check List

Tests

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-tests-checked size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Dec 31, 2024
Copy link

ti-chi-bot bot commented Dec 31, 2024

Welcome @chibiegg!

It looks like this is your first PR to pingcap/tidb 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to pingcap/tidb. 😃

Copy link

ti-chi-bot bot commented Dec 31, 2024

Hi @chibiegg. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copy link

tiprow bot commented Dec 31, 2024

Hi @chibiegg. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@@ -798,120 +800,137 @@ func (sc *StatementContext) AddAffectedRows(rows uint64) {
// For compatibility with MySQL, not add the affected row cause by the foreign key trigger.
return
}
sc.mu.Lock()
defer sc.mu.Unlock()
mu := &sc.mu
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't get the point that why using this way can avoid panic, any reference for mentioning it? From my persepctive, the panic may be related to other places.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn’t fully aware of the direct cause at that point. Your observation is indeed correct.
The direct cause was the Reset() function, which replaces sc itself, and consequently replaces mu.

As mentioned in this comment, how about locking the Mutex in Reset()?

#58600 (comment)

@Defined2014 Defined2014 added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. do-not-merge/needs-tests-checked labels Dec 31, 2024
Copy link

codecov bot commented Dec 31, 2024

Codecov Report

Attention: Patch coverage is 33.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 73.5141%. Comparing base (42d4fae) to head (18b12a2).
Report is 28 commits behind head on master.

Additional details and impacted files
@@               Coverage Diff                @@
##             master     #58629        +/-   ##
================================================
+ Coverage   73.1093%   73.5141%   +0.4047%     
================================================
  Files          1676       1677         +1     
  Lines        463354     468929      +5575     
================================================
+ Hits         338755     344729      +5974     
+ Misses       103788     103317       -471     
- Partials      20811      20883        +72     
Flag Coverage Δ
integration 43.0288% <33.3333%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.6910% <ø> (ø)
parser ∅ <ø> (∅)
br 45.0039% <ø> (-0.7985%) ⬇️

@chibiegg
Copy link
Author

chibiegg commented Jan 1, 2025

I’ve applied the patch suggested in the Issue comment and have been testing it in my environment.
I did not apply the Mutex patch in this commit; instead, I only applied the patch that modifies setDataForProcessList to handle RefCountOfStmtCtx .

I’m not certain if this is the only fix needed, but so far, the issue hasn’t recurred.

I’ll keep observing it for a while.

@ti-chi-bot ti-chi-bot bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 6, 2025
@chibiegg
Copy link
Author

chibiegg commented Jan 6, 2025

@Defined2014 Thank you for the suggestion. I've updated this PR to reflect the changes that I’ve already tested and confirmed.

Copy link

ti-chi-bot bot commented Jan 6, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Defined2014

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 6, 2025
Copy link

ti-chi-bot bot commented Jan 6, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-01-06 06:34:17.484522143 +0000 UTC m=+162600.773353845: ☑️ agreed by Defined2014.

@chibiegg
Copy link
Author

chibiegg commented Jan 6, 2025

/test fast_test_tiprow

Copy link

ti-chi-bot bot commented Jan 6, 2025

@chibiegg: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-integration-e2e-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-scan-deps
/test pull-sqllogic-test
/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_integration_e2e_test
pingcap/tidb/pull_mysql_client_test

In response to this:

/test fast_test_tiprow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copy link

tiprow bot commented Jan 6, 2025

@ti-chi-bot[bot]: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test fast_test_tiprow
  • /test tidb_parser_test

Use /test all to run all jobs.

In response to this:

@chibiegg: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-integration-e2e-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-scan-deps
/test pull-sqllogic-test
/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_integration_e2e_test
pingcap/tidb/pull_mysql_client_test

In response to this:

/test fast_test_tiprow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@chibiegg
Copy link
Author

chibiegg commented Jan 6, 2025

/retest-required

Copy link

tiprow bot commented Jan 6, 2025

@chibiegg: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
fast_test_tiprow 18b12a2 link true /test fast_test_tiprow

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copy link

ti-chi-bot bot commented Jan 6, 2025

@chibiegg: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
idc-jenkins-ci-tidb/unit-test 18b12a2 link true /test unit-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@chibiegg
Copy link
Author

chibiegg commented Jan 6, 2025

I received a notification from ti-chi-bot that some tests have failed.
Is there anything I need to do?

I confirmed that make passes successfully on my local environment.

@Defined2014
Copy link
Contributor

Defined2014 commented Jan 6, 2025

I received a notification from ti-chi-bot that some tests have failed. Is there anything I need to do?

I confirmed that make passes successfully on my local environment.

Seems TestSomeTables test case failed. Maybe we could move Increase() and Decrease() lated logical into ToRow function.

@@ -1839,7 +1839,11 @@ func (e *memtableRetriever) setDataForProcessList(ctx sessionctx.Context) {
continue
}

if pi.StmtCtx != nil && pi.RefCountOfStmtCtx != nil && !pi.RefCountOfStmtCtx.TryIncrease() {
continue
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Directly using continue here might lead to the loss of some row data. It could be a better approach to perform the check within ProcessInfo.ToRow

diff --git a/pkg/util/processinfo.go b/pkg/util/processinfo.go
index 92559da329..f8a3143ed0 100644
--- a/pkg/util/processinfo.go
+++ b/pkg/util/processinfo.go
@@ -137,7 +137,10 @@ func (pi *ProcessInfo) txnStartTs(tz *time.Location) (txnStart string) {
 func (pi *ProcessInfo) ToRow(tz *time.Location) []any {
        bytesConsumed := int64(0)
        diskConsumed := int64(0)
-       if pi.StmtCtx != nil {
+       var affectedRows any
+       var cpuUsages ppcpuusage.CPUUsages
+       if pi.StmtCtx != nil && pi.RefCountOfStmtCtx != nil && !pi.RefCountOfStmtCtx.TryIncrease(){
+               affectedRows = pi.StmtCtx.AffectedRows()
                if pi.MemTracker != nil {
                        bytesConsumed = pi.MemTracker.BytesConsumed()
                }
@@ -146,11 +149,6 @@ func (pi *ProcessInfo) ToRow(tz *time.Location) []any {
                }
        }
 
-       var affectedRows any
-       var cpuUsages ppcpuusage.CPUUsages
-       if pi.StmtCtx != nil {
-               affectedRows = pi.StmtCtx.AffectedRows()
-       }
        if pi.SQLCPUUsage != nil {
                cpuUsages = pi.SQLCPUUsage.GetCPUUsages()
        }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. ok-to-test Indicates a PR is ready to be tested. release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

fatal error: sync: unlock of unlocked mutex (v8.5.0 stmtctx.go:803)
4 participants