roachtest: kv/gracefuldraining failed #130760

cockroach-teamcity · 2024-09-15T07:17:11Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 3542ba5a3f47f9042e81583fe52c893aac9c4e3c:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:604
	            				main/pkg/cmd/roachtest/monitor.go:120
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1687.00 at time 2024-09-15 07:16:06.213452043 +0000 UTC m=+5283.522635291 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:587
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:604
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:120
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1687.00 at time 2024-09-15 07:16:06.213452043 +0000 UTC m=+5283.522635291 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2355).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #130747 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv branch-master release-blocker]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!

Jira issue: CRDB-42211}

The text was updated successfully, but these errors were encountered:

tbg · 2024-09-16T13:44:29Z

This is like #130747 - right at the beginning of the test, when we take a node down gracefully, throughput takes a hit.

130796: roachtest: improve observability in kv/gracefuldraining r=andrewbaptist a=tbg This test sometimes fails since QPS tanks in the aftermath of a gracefully shut down node. This commit adds a one-minute warm-up period during which stats are collected and sets up statement bundle collection. The hope is that future failures will have complete traces of a number of failed requests, which should make it easy to understand why throughput was adversely affected. Fixes #130747. Touches #130760 (release-24.2; needs backport) Release note: None Epic: None Co-authored-by: Tobias Grieger <[email protected]>

cockroach-teamcity · 2024-09-17T11:25:13Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 7a32a78a1f7a691f32a131d79f6ae00a19e20e86:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:604
	            				main/pkg/cmd/roachtest/monitor.go:120
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1769.00 at time 2024-09-17 11:24:14.939177309 +0000 UTC m=+20233.357653786 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:587
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:604
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:120
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1769.00 at time 2024-09-17 11:24:14.939177309 +0000 UTC m=+20233.357653786 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2355).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-09-19T11:29:17Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 6151abc4225aecfe5f5fc4655b619d560226777a:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:604
	            				main/pkg/cmd/roachtest/monitor.go:120
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1664.00 at time 2024-09-19 11:28:14.341313544 +0000 UTC m=+20403.903009326 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:587
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:604
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:120
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1664.00 at time 2024-09-19 11:28:14.341313544 +0000 UTC m=+20403.903009326 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2355).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-09-23T10:51:05Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ a245c3f2811e01c9d51eae75612ccd5a6d406464:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:604
	            				main/pkg/cmd/roachtest/monitor.go:120
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1603.00 at time 2024-09-23 10:49:52.405886303 +0000 UTC m=+18105.132132282 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:587
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:604
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:120
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1603.00 at time 2024-09-23 10:49:52.405886303 +0000 UTC m=+18105.132132282 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2355).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

kvoli · 2024-10-03T16:51:51Z

I'm going to remove my assignment -- as this involvement was primarily triage related. If the test fails again with the CPU profiles available, I'll take a look.

cockroach-teamcity · 2024-11-02T11:53:43Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 2b75ba5e0a0793b946e8ae43ddf671cbd3ea1a5f:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 835.00 at time 2024-11-02 11:52:18.476106031 +0000 UTC m=+22227.836751161 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 835.00 at time 2024-11-02 11:52:18.476106031 +0000 UTC m=+22227.836751161 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2331).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2.5-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-11-03T09:02:26Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 134112c2d6920bd50d93d1d73596a34a477070b6:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1607.00 at time 2024-11-03 08:01:03.366759821 +0000 UTC m=+4446.196971672 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1607.00 at time 2024-11-03 08:01:03.366759821 +0000 UTC m=+4446.196971672 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2331).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2.5-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-11-06T08:27:43Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 93b67a9956891893444de8e20eb2a22c5ad55624:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1596.00 at time 2024-11-06 08:26:15.659625451 +0000 UTC m=+6103.807473896 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1596.00 at time 2024-11-06 08:26:15.659625451 +0000 UTC m=+6103.807473896 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2331).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2.5-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-11-07T08:37:59Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 2112e82243d0f454e659236dbf76937ce73dc013:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1198.00 at time 2024-11-07 08:36:37.005556735 +0000 UTC m=+6921.650801467 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1198.00 at time 2024-11-07 08:36:37.005556735 +0000 UTC m=+6921.650801467 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2331).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2.5-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-11-08T08:28:44Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ c9c36aba9df73ce233967020efd5b96e1992378a:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 835.00 at time 2024-11-08 08:27:21.132817327 +0000 UTC m=+6240.006711291 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 835.00 at time 2024-11-08 08:27:21.132817327 +0000 UTC m=+6240.006711291 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2331).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2.5-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-11-09T12:18:30Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 5764366875adca70c3a3ff3d4d13aaf599bc53a7:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1687.00 at time 2024-11-09 12:17:09.30540077 +0000 UTC m=+20110.136341094 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1687.00 at time 2024-11-09 12:17:09.30540077 +0000 UTC m=+20110.136341094 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2331).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2.5-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-11-10T08:39:43Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 5764366875adca70c3a3ff3d4d13aaf599bc53a7:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1242.00 at time 2024-11-10 08:38:14.153434643 +0000 UTC m=+7019.213714928 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1242.00 at time 2024-11-10 08:38:14.153434643 +0000 UTC m=+7019.213714928 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2331).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2.5-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-11-12T08:38:22Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ be69012d37da70d0ada6eb4cf2b008d02ef16ffd:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1051.00 at time 2024-11-12 08:37:03.536936651 +0000 UTC m=+6503.579106231 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1051.00 at time 2024-11-12 08:37:03.536936651 +0000 UTC m=+6503.579106231 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2331).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2.5-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-11-13T12:04:32Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ c9b7151258520849effaa3abd9514f00fb240039:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1668.00 at time 2024-11-13 12:03:11.222685544 +0000 UTC m=+18929.536325422 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1668.00 at time 2024-11-13 12:03:11.222685544 +0000 UTC m=+18929.536325422 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2331).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2.5-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-11-22T08:22:11Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 1ba69a2843ef6f3ffdd8a3863de571342aa501eb:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1725.00 at time 2024-11-22 08:21:03.6268154 +0000 UTC m=+5354.792970685 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1725.00 at time 2024-11-22 08:21:03.6268154 +0000 UTC m=+5354.792970685 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #135161 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-24.3.0-rc]
roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-11-28T12:31:14Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ fdb6c4cd2758a1b915e5815bce30dde3616b7ff9:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1703.00 at time 2024-11-28 12:29:59.96034127 +0000 UTC m=+20576.534685932 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1703.00 at time 2024-11-28 12:29:59.96034127 +0000 UTC m=+20576.534685932 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
fs=ext4
localSSD=true
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #136353 roachtest: kv/gracefuldraining failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv branch-release-24.3.1-rc release-blocker]
roachtest: kv/gracefuldraining failed #135161 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-24.3.0-rc]
roachtest: kv/gracefuldraining failed #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-12-11T08:11:59Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ b7aadefdfe99a5ca617a6d73c14915a494be8128:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1770.00 at time 2024-12-11 08:10:39.344604618 +0000 UTC m=+4762.092171758 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1770.00 at time 2024-12-11 08:10:39.344604618 +0000 UTC m=+4762.092171758 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #136353 roachtest: kv/gracefuldraining failed [A-testing B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.3.1-rc]
roachtest: kv/gracefuldraining failed #135161 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-24.3.0-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2025-01-08T08:19:52Z

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 3428b140d7f5019b89e37164ca6c2fb3af981727:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1783.00 at time 2025-01-08 08:18:48.260678774 +0000 UTC m=+5487.233109589 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:622
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1783.00 at time 2025-01-08 08:18:48.260678774 +0000 UTC m=+5487.233109589 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: kv/gracefuldraining failed #138116 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.8-rc]
roachtest: kv/gracefuldraining failed #137970 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.3.3-rc]
roachtest: kv/gracefuldraining failed #136353 roachtest: kv/gracefuldraining failed [A-testing B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.3.1-rc]
roachtest: kv/gracefuldraining failed #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
roachtest: kv/gracefuldraining failed #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

_{This test on roachdash | Improve this report!}

arulajmani · 2025-01-21T15:45:24Z

Closing now that the metrics interval PR has been backported. #131569 (comment).

tbg added X-duplicate Closed as a duplicate of another issue. and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 16, 2024

tbg mentioned this issue Sep 16, 2024

roachtest: improve observability in kv/gracefuldraining #130796

Merged

blathers-crl bot mentioned this issue Sep 16, 2024

release-24.2: roachtest: improve observability in kv/gracefuldraining #130806

Merged

cockroach-teamcity mentioned this issue Sep 29, 2024

roachtest: kv/gracefuldraining failed #131569

Closed

arulajmani assigned kvoli Sep 30, 2024

arulajmani added the P-3 Issues/test failures with no fix SLA label Sep 30, 2024

kvoli removed their assignment Oct 3, 2024

cockroach-teamcity mentioned this issue Oct 20, 2024

roachtest: kv/gracefuldraining failed #133020

Closed

cockroach-teamcity mentioned this issue Nov 1, 2024

roachtest: kv/gracefuldraining failed #134044

Closed

cockroach-teamcity mentioned this issue Nov 14, 2024

roachtest: kv/gracefuldraining failed #135161

Closed

cockroach-teamcity mentioned this issue Nov 28, 2024

roachtest: kv/gracefuldraining failed #136353

Closed

This was referenced Dec 24, 2024

roachtest: kv/gracefuldraining failed #137970

Closed

roachtest: kv/gracefuldraining failed #138116

Closed

arulajmani self-assigned this Jan 15, 2025

exalate-issue-sync bot unassigned arulajmani Jan 15, 2025

arulajmani closed this as completed Jan 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: kv/gracefuldraining failed #130760

roachtest: kv/gracefuldraining failed #130760

cockroach-teamcity commented Sep 15, 2024 •

edited by cockroach-jira-scripts

Loading

tbg commented Sep 16, 2024

cockroach-teamcity commented Sep 17, 2024

cockroach-teamcity commented Sep 19, 2024

cockroach-teamcity commented Sep 23, 2024

kvoli commented Oct 3, 2024

cockroach-teamcity commented Nov 2, 2024

cockroach-teamcity commented Nov 3, 2024

cockroach-teamcity commented Nov 6, 2024

cockroach-teamcity commented Nov 7, 2024

cockroach-teamcity commented Nov 8, 2024

cockroach-teamcity commented Nov 9, 2024

cockroach-teamcity commented Nov 10, 2024

cockroach-teamcity commented Nov 12, 2024

cockroach-teamcity commented Nov 13, 2024

cockroach-teamcity commented Nov 22, 2024

cockroach-teamcity commented Nov 28, 2024

cockroach-teamcity commented Dec 11, 2024

cockroach-teamcity commented Jan 8, 2025

arulajmani commented Jan 21, 2025

roachtest: kv/gracefuldraining failed #130760

roachtest: kv/gracefuldraining failed #130760

Comments

cockroach-teamcity commented Sep 15, 2024 • edited by cockroach-jira-scripts Loading

tbg commented Sep 16, 2024

cockroach-teamcity commented Sep 17, 2024

cockroach-teamcity commented Sep 19, 2024

cockroach-teamcity commented Sep 23, 2024

kvoli commented Oct 3, 2024

cockroach-teamcity commented Nov 2, 2024

cockroach-teamcity commented Nov 3, 2024

cockroach-teamcity commented Nov 6, 2024

cockroach-teamcity commented Nov 7, 2024

cockroach-teamcity commented Nov 8, 2024

cockroach-teamcity commented Nov 9, 2024

cockroach-teamcity commented Nov 10, 2024

cockroach-teamcity commented Nov 12, 2024

cockroach-teamcity commented Nov 13, 2024

cockroach-teamcity commented Nov 22, 2024

cockroach-teamcity commented Nov 28, 2024

cockroach-teamcity commented Dec 11, 2024

cockroach-teamcity commented Jan 8, 2025

arulajmani commented Jan 21, 2025

cockroach-teamcity commented Sep 15, 2024 •

edited by cockroach-jira-scripts

Loading