roachtest: kv/gracefuldraining failed #133020
Assignees
Labels
branch-release-24.3
Used to mark GA and release blockers, technical advisories, and bugs for 24.3
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
P-2
Issues/test failures with a fix SLA of 3 months
T-kv
KV Team
Comments
cockroach-teamcity
added
branch-release-24.3
|
Duplicate of #131569; Copying over labels. |
arulajmani
added
C-bug
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 330f659a3f244d1386bb86c89a258d2da6063959: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 416acdb7560e4548f3b82c5a522ef48592471c7d: Parameters:
Same failure on other branches
|
|
Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout. roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ c357ebbfdd72bb6aecef57c84850845b5955382f: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ b1fbd0c00f283c147b0c3a36a8f25ce7eec41dff: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ c077ebf6e98bcd579481b93c83f14184ab94f2e6: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 29d723d5f5e6ed2075feb728e9632fb3516e4fc2: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ bbce415047c9896ee3b33b1eb4c06e3d2cab5bd6: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ bbce415047c9896ee3b33b1eb4c06e3d2cab5bd6: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ cb680f5c1811b890ecde369ec5e7d18e05ef1b37: Parameters:
Same failure on other branches
|
|
Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout. roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 7252771de42bf0752052fae4f945c0e7a88d0ecd: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 093d4f3eb760cb5cb5cc52679307351cf6f46a45: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 16442333a9c36fab40e2223c6b33c22be0c97b5f: Parameters:
Same failure on other branches
|
|
Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout. roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 16442333a9c36fab40e2223c6b33c22be0c97b5f: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 58b09793ce98dfaaefbe3577d19703931a781181: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ bca9d708828f2e202df23b343efb8119a610c030: Parameters:
Same failure on other branches
|
|
Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout. roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ d11f59a637b473482a2186ccdda104519689b919: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ ddc9e8fc1147f9b998f99da82231b06175eb6aa6: Parameters:
Same failure on other branches
|
|
Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout. roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 7951487513959ad707472f7f9231b0a2e3ad067b: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 7951487513959ad707472f7f9231b0a2e3ad067b: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ db41e1ca7a3e66e9b9bde253585dd406e518d0e9: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ ef0e25554c256e3c014d1588a526f96258f91e95: Parameters:
Same failure on other branches
|
|
Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout. roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 0f54637e66126ddfe019eac34afc8de303f89f9c: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 95873120fc094132a8717e11d2eb919889bc05ea: Parameters:
Same failure on other branches
|
|
The latest failure fails with: But the SQL graph shows QPS of ~2,000 throughout. 
Something's not adding up here. |
|
Seeing something similar here as well: #133020 (comment). I wonder if this is because we're collecting metrics over 1 second intervals. Before cf30717, we used to collect QPS metrics over a 10s interval. There's some words about the impact of this interval here: cockroach/pkg/cmd/roachtest/roachtestutil/profile.go Lines 238 to 241 in cb42e40 The timelines of this test starting to fail and cf30717 don't line up though, so that's a mystery. |
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 16, 2025
The gracefuldrain test was modernized in cf30717. Prior to that commit, QPS metrics were collected over a 10s interval, whereas the modernization refactor changed this to 1 second intervals. Looking at a few recent test failures, I see QPS metrics above the failure threshold, which makes me think suspect that this 1s interval is causing the sorts of inaccuracies MeasureQPS warns against. Also See cockroachdb#133020 (comment). One thing that doesn't line up is the timeline of this tests failure and cf30717. Still, this patch changes the metric's interval back to 10s. References cockroachdb#133020 Release note: None
|
Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout. roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ c12361f072e8495b5401ec93e732bd5830598395: Parameters:
Same failure on other branches
|
craig bot
pushed a commit
that referenced
this issue
Jan 17, 2025
139062: server,sql: increase redaction coverage of diagnostics tests r=angles-n-daemons a=dhartunian This change adds test coverage to the diagnostic reporter that's meant to catch situations where schema or statement scrubbing is accidentally turned off. In the course of adding tests for SQL Stats it was discovered that the diagnostics reporter would include statements that were in internal applications (`$ internal` prefix) so a change was made to omit those from the reports. Resolves: #134450 Release note: None 139066: db-console: update rac v2 overload dashboard charts r=sumeerbhola a=kvoli Update the db console overload dashboard to: - remove metrics associated with v1 replication admission control - rename metrics associated with v2 replication admission control to remove the version reference - add a chart containing the per-node send queue size in bytes <details><summary>Screenshots</summary> <p>    </p> </details> Resolves: #128039 Release note (ui change): The overload dashboard on DB Console now shows only the v2 replication admission control metrics, where previously it displayed both v1 and v2 metrics. Additionally, the aggregate size of queued replication entries is now shown. 139171: sql: use parsed statements for persistedsqlstats r=fqazi a=fqazi Previously, we would re-parse SQL statements used to upsert statement and txn stats. To address this patch, this patch will parse these statements once and use ExecParsed to reduce CPU usage. This patch also adds a simple benchmark for this code path as well, which shows a small 1% delta. Before: BenchmarkSQLStatsFlush 100 1415926687 ns/op 319339313 B/op 2302002 allocs/op After: BenchmarkSQLStatsFlush 100 1396673170 ns/op 319003310 B/op 2298192 allocs/op Fixes: #134583 Release note: None 139273: roachtest: collect qps metrics over longer window in gracefuldrain test r=arulajmani a=arulajmani The gracefuldrain test was modernized in cf30717. Prior to that commit, QPS metrics were collected over a 10s interval, whereas the modernization refactor changed this to 1 second intervals. Looking at a few recent test failures, I see QPS metrics above the failure threshold, which makes me think suspect that this 1s interval is causing the sorts of inaccuracies MeasureQPS warns against. Also See #133020 (comment). One thing that doesn't line up is the timeline of this tests failure and cf30717. Still, this patch changes the metric's interval back to 10s. References #133020 Release note: None Co-authored-by: David Hartunian <[email protected]> Co-authored-by: Austen McClernon <[email protected]> Co-authored-by: Faizan Qazi <[email protected]> Co-authored-by: Arul Ajmani <[email protected]>
For the recent test failure: I've growing conviction this is because of the metrics interval window issue. |
blathers-crl bot
pushed a commit
that referenced
this issue
Jan 17, 2025
The gracefuldrain test was modernized in cf30717. Prior to that commit, QPS metrics were collected over a 10s interval, whereas the modernization refactor changed this to 1 second intervals. Looking at a few recent test failures, I see QPS metrics above the failure threshold, which makes me think suspect that this 1s interval is causing the sorts of inaccuracies MeasureQPS warns against. Also See #133020 (comment). One thing that doesn't line up is the timeline of this tests failure and cf30717. Still, this patch changes the metric's interval back to 10s. References #133020 Release note: None
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 1cce4f247805c3725e74d89395d4c937e6b6408e: Parameters:
Same failure on other branches
|
|
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ 1cce4f247805c3725e74d89395d4c937e6b6408e: Parameters:
Same failure on other branches
|
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 20, 2025
The gracefuldrain test was modernized in cf30717. Prior to that commit, QPS metrics were collected over a 10s interval, whereas the modernization refactor changed this to 1 second intervals. Looking at a few recent test failures, I see QPS metrics above the failure threshold, which makes me think suspect that this 1s interval is causing the sorts of inaccuracies MeasureQPS warns against. Also See cockroachdb#133020 (comment). One thing that doesn't line up is the timeline of this tests failure and cf30717. Still, this patch changes the metric's interval back to 10s. References cockroachdb#133020 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 20, 2025
The gracefuldrain test was modernized in cf30717. Prior to that commit, QPS metrics were collected over a 10s interval, whereas the modernization refactor changed this to 1 second intervals. Looking at a few recent test failures, I see QPS metrics above the failure threshold, which makes me think suspect that this 1s interval is causing the sorts of inaccuracies MeasureQPS warns against. Also See cockroachdb#133020 (comment). One thing that doesn't line up is the timeline of this tests failure and cf30717. Still, this patch changes the metric's interval back to 10s. References cockroachdb#133020 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 20, 2025
The gracefuldrain test was modernized in cf30717. Prior to that commit, QPS metrics were collected over a 10s interval, whereas the modernization refactor changed this to 1 second intervals. Looking at a few recent test failures, I see QPS metrics above the failure threshold, which makes me think suspect that this 1s interval is causing the sorts of inaccuracies MeasureQPS warns against. Also See cockroachdb#133020 (comment). One thing that doesn't line up is the timeline of this tests failure and cf30717. Still, this patch changes the metric's interval back to 10s. References cockroachdb#133020 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 20, 2025
The gracefuldrain test was modernized in cf30717. Prior to that commit, QPS metrics were collected over a 10s interval, whereas the modernization refactor changed this to 1 second intervals. Looking at a few recent test failures, I see QPS metrics above the failure threshold, which makes me think suspect that this 1s interval is causing the sorts of inaccuracies MeasureQPS warns against. Also See cockroachdb#133020 (comment). One thing that doesn't line up is the timeline of this tests failure and cf30717. Still, this patch changes the metric's interval back to 10s. References cockroachdb#133020 Release note: None
|
Closing since the metrics interval PR has been backported. Also see: #131569 (comment). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
roachtest.kv/gracefuldraining failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_runtimeAssertionsBuild=falseROACHTEST_ssd=0Help
See: roachtest README
See: How To Investigate (internal)
See: Grafana
Same failure on other branches
This test on roachdash | Improve this report!
Jira issue: CRDB-43414
The text was updated successfully, but these errors were encountered: