Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: kv/gracefuldraining failed #135161

Closed
cockroach-teamcity opened this issue Nov 14, 2024 · 10 comments
Closed

roachtest: kv/gracefuldraining failed #135161

cockroach-teamcity opened this issue Nov 14, 2024 · 10 comments
Labels
A-testing Testing tools and infrastructure branch-release-24.3.0-rc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). no-test-failure-activity O-roachtest O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team X-stale

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 14, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.kv/gracefuldraining failed with artifacts on release-24.3.0-rc @ 516de012609eb9463b2df0fdc78958ca0549074e:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:623
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1792.00 at time 2024-11-14 11:38:39.835092719 +0000 UTC m=+17852.682065753 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1792.00 at time 2024-11-14 11:38:39.835092719 +0000 UTC m=+17852.682065753 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2451).Run: context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-44382
@cockroach-teamcity cockroach-teamcity added branch-release-24.3.0-rc C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Nov 14, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 15, 2024
Closed
@tbg
Copy link
Member

tbg commented Nov 15, 2024 

_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
   62.0s        0          545.1          487.5      1.0      1.4      1.6      1.8 follower-read
   62.0s        0          479.1          485.0      1.0      1.5      1.7      4.7 read
   62.0s        0          974.3          974.4      2.2     37.7     48.2     67.1 write

   63.0s        0          496.0          487.7      1.0      1.4      1.5      1.8 follower-read
   63.0s        0          484.0          485.0      1.0      1.4      1.6      1.7 read
   63.0s        0         1021.0          975.2      2.2     17.8     25.2     28.3 write

   64.0s        0          519.7          488.2      1.0      1.4      1.6     10.0 follower-read
   64.0s        0          510.7          485.4      1.0      1.6      2.0      6.8 read
   64.0s        0          969.4          975.1      2.4     22.0     26.2     30.4 write

   65.0s        0          482.3          488.1      1.0      1.5      1.6      2.0 follower-read
   65.0s        0          510.4          485.8      1.0      1.5      1.6      2.4 read
   65.0s        0         1008.7          975.6      2.1     10.5     23.1     27.3 write

I don't see it, this looks fine (in an ideal world, write is 1000, and read and follower-read are 500). Removing rel-blocker.

@tbg tbg added P-3 Issues/test failures with no fix SLA C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Nov 15, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 15, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on release-24.3.0-rc @ 49d902f212d10c8482298d777061ba153d5c16c9:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:623
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1784.00 at time 2024-11-16 08:21:23.4625686 +0000 UTC m=+5944.480475733 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1784.00 at time 2024-11-16 08:21:23.4625686 +0000 UTC m=+5944.480475733 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv/gracefuldraining failed with artifacts on release-24.3.0-rc @ 49d902f212d10c8482298d777061ba153d5c16c9:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:623
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1770.00 at time 2024-11-17 11:35:06.591083495 +0000 UTC m=+17612.251907031 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1770.00 at time 2024-11-17 11:35:06.591083495 +0000 UTC m=+17612.251907031 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2451).Run: context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv/gracefuldraining failed with artifacts on release-24.3.0-rc @ ef2ebe96f3a26fd6d6716e2ec4a0c8c6fe0fb567:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:623
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1693.00 at time 2024-11-21 11:28:30.682778565 +0000 UTC m=+17085.431442548 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1693.00 at time 2024-11-21 11:28:30.682778565 +0000 UTC m=+17085.431442548 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 22, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.kv/gracefuldraining failed with artifacts on release-24.3.0-rc @ ef2ebe96f3a26fd6d6716e2ec4a0c8c6fe0fb567:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:623
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1726.00 at time 2024-11-23 11:27:42.467422159 +0000 UTC m=+17216.971653362 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1726.00 at time 2024-11-23 11:27:42.467422159 +0000 UTC m=+17216.971653362 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv/gracefuldraining failed with artifacts on release-24.3.0-rc @ 330209cefef4383a9d67ce7d86e2d3a9794ccd4f:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:623
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1580.00 at time 2024-11-27 11:28:51.67258369 +0000 UTC m=+17152.986904190 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1580.00 at time 2024-11-27 11:28:51.67258369 +0000 UTC m=+17152.986904190 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 28, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on release-24.3.0-rc @ 330209cefef4383a9d67ce7d86e2d3a9794ccd4f:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:623
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1742.00 at time 2024-11-28 11:04:52.900041688 +0000 UTC m=+16064.226448716 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1742.00 at time 2024-11-28 11:04:52.900041688 +0000 UTC m=+16064.226448716 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv/gracefuldraining failed with artifacts on release-24.3.0-rc @ 330209cefef4383a9d67ce7d86e2d3a9794ccd4f:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:623
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1739.00 at time 2024-11-30 07:54:23.590948394 +0000 UTC m=+4253.074550070 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1739.00 at time 2024-11-30 07:54:23.590948394 +0000 UTC m=+4253.074550070 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv/gracefuldraining failed with artifacts on release-24.3.0-rc @ 330209cefef4383a9d67ce7d86e2d3a9794ccd4f:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:623
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1582.00 at time 2024-12-01 10:45:57.967109077 +0000 UTC m=+14705.136841013 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1582.00 at time 2024-12-01 10:45:57.967109077 +0000 UTC m=+14705.136841013 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Dec 24, 2024
Closed
Closed
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 1, 2025

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

@github-actions github-actions bot closed this as completed Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-testing Testing tools and infrastructure branch-release-24.3.0-rc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). no-test-failure-activity O-roachtest O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team X-stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tbg @cockroach-teamcity