Occasional performance regression on benchmarksql

[owlsome2501]

Bug Report

There is occasional performance regression on benchmarksql.
releated issue: #55426

1. Minimal reproduce step (Required)

1. Deploy a cluster with 3 TiDB and 3 Tikv.
2. Run benchmark benchmarksql (benchmarksql-1k) multiple times until find the specific io or cpu usage pattern on grafana.

2. What did you expect to see? (Required)

No performance regression.
QPS: 64039 (+6.4% compared to baseline 60193)

or
QPS: 61259 (+1.77% compared to baseline 60193)

3. What did you see instead (Required)

Performance regression .
QPS: 55187 (-8.3% compared to baseline 60193)

or
QPS: 54639 (-9.20% compared to baseline 60193)

4. What is your TiDB version? (Required)

26443da
tikv/tikv@7c61278
tikv/pd@098b802
or
74034d4
tikv/tikv@de06b38
tikv/pd@76bc34a

[owlsome2501]

/severity major
/label affects-8.3
/label affects-8.4

[ti-chi-bot]

@owlsome2501: The label(s) affects-8.3 cannot be applied. These labels are supported: fuzz/sqlancer, challenge-program, compatibility-breaker, first-time-contributor, contribution, good first issue, correctness, duplicate, proposal, security, needs-more-info, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, needs-cherry-pick-release-8.1, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.4, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5, may-affects-8.1.

In response to this:

/severity major
/label affects-8.3
/label affects-8.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[jebter]

@owlsome2501 Can you confirm which modules introduced the issue?

[owlsome2501]

Unable to determine the module. It may be hot spots or load balancing problem.