roachtest: restoreTPCCInc/nodes=10 failed #87331

cockroach-teamcity · 2022-09-02T13:45:27Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ c080f7ac2f5be11aedd787dceba77cca7df18a16:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:453,test_runner.go:897: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:453
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6240
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1028,test_runner.go:927: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]
roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1 release-blocker]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!

Jira issue: CRDB-19276}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-09-03T13:44:04Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ 4dcb32c0346e20a95847763f89b9b0796d9ed4dc:

		  | * go.opencensus.io/stats/view.(*worker).start(0xc000bf9080)
		  | * 	go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:276 +0xad
		  | * created by go.opencensus.io/stats/view.init.0
		  | * 	go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:34 +0x8d
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:453,test_runner.go:897: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:453
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6240
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1028,test_runner.go:927: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]
roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-04T19:50:32Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ b0f13c9bbef3e6628471d887672be7c7658f6511:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:453,test_runner.go:897: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:453
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6240
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1028,test_runner.go:927: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]
roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-05T13:25:51Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ e5917605f751c569ddd90062250a40cafd7ccf30:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:453,test_runner.go:897: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:453
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6240
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1028,test_runner.go:927: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]
roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-06T13:39:11Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ 6ffaa1d907b7c9194df4ffff412f48a63bf432df:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:453,test_runner.go:897: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:453
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6240
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1028,test_runner.go:927: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]
roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1 release-blocker]

_{This test on roachdash | Improve this report!}

stevendanna · 2022-09-07T12:18:51Z

I currently suspect that this is related to #86496, just based on timing.

In the most recent run, the backup had progressed less than 1% according to the progress in the job. The CPU stack in the debug zip shows that our nodes are still doing work, with cpu profiles that look like:

I'll note that the backup in this case has 83000 entries returned from SHOW BACKUP FILES FROM.

cockroach-teamcity · 2022-09-07T20:27:08Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ 2372698da1dfacb90f60c6a63f2c1298d1db16b8:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:453,test_runner.go:897: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:453
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6240
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1028,test_runner.go:927: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]
roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-08T13:49:14Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ e39111b2e714375faa0facc05e51e8f619a55b21:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:453,test_runner.go:908: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:453
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:908
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1039,test_runner.go:938: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #87602 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.2 release-blocker]
roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]
roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-09T19:27:46Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ a82711442c65cf14489c55041b45b11a1e38415b:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:474,test_runner.go:906: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:474
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:906
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1037,test_runner.go:936: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #87602 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.2 release-blocker]
roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]
roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-10T19:30:54Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ 389661e823c19f318fa07ec2278336262531692d:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:474,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:474
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1048,test_runner.go:947: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #87602 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.2 release-blocker]
roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]
roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-11T19:35:52Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ bc2e47da0523b347c28cf024707e80cd35d6c98a:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:474,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:474
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1048,test_runner.go:947: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #87602 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.2 release-blocker]
roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-12T20:16:09Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ 773568fbda06ba9be9fb1bc34a331f21c8891ffa:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:474,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:474
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1048,test_runner.go:947: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #87602 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio blocks-22.2.0-beta.1 branch-release-22.2 release-blocker]
roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-13T13:36:08Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on master @ 2cfa6b1779ead01508c04e228bf72b4e0e96d98c:

		  | * github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xa5
		  | * created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | * 	github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1ed
		  | * 
		  | *
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach sql --insecure -e "
		  | 				RESTORE FROM '2021/05/21-020411.00' IN
		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,restore.go:474,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:474
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1048,test_runner.go:947: test timed out (6h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #87602 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio blocks-22.2.0-beta.1 branch-release-22.2 release-blocker]
roachtest: restoreTPCCInc/nodes=10 failed #85452 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]

_{This test on roachdash | Improve this report!}

dt · 2022-09-14T16:51:33Z

#87907

cockroach-teamcity added this to the 22.2 milestone Sep 2, 2022

blathers-crl bot added the T-disaster-recovery label Sep 2, 2022

cockroach-teamcity mentioned this issue Sep 3, 2022

roachtest: restoreTPCCInc/nodes=10 failed #79030

Closed

celiala added the blocks-22.2.0-beta.1 label Sep 6, 2022

cockroach-teamcity mentioned this issue Sep 8, 2022

roachtest: restoreTPCCInc/nodes=10 failed #87602

Closed

dt closed this as completed Sep 14, 2022

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: restoreTPCCInc/nodes=10 failed #87331

roachtest: restoreTPCCInc/nodes=10 failed #87331

cockroach-teamcity commented Sep 2, 2022 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Sep 3, 2022

cockroach-teamcity commented Sep 4, 2022

cockroach-teamcity commented Sep 5, 2022

cockroach-teamcity commented Sep 6, 2022

stevendanna commented Sep 7, 2022 •

edited

Loading

cockroach-teamcity commented Sep 7, 2022

cockroach-teamcity commented Sep 8, 2022

cockroach-teamcity commented Sep 9, 2022

cockroach-teamcity commented Sep 10, 2022

cockroach-teamcity commented Sep 11, 2022

cockroach-teamcity commented Sep 12, 2022

cockroach-teamcity commented Sep 13, 2022

dt commented Sep 14, 2022

roachtest: restoreTPCCInc/nodes=10 failed #87331

roachtest: restoreTPCCInc/nodes=10 failed #87331

Comments

cockroach-teamcity commented Sep 2, 2022 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Sep 3, 2022

cockroach-teamcity commented Sep 4, 2022

cockroach-teamcity commented Sep 5, 2022

cockroach-teamcity commented Sep 6, 2022

stevendanna commented Sep 7, 2022 • edited Loading

cockroach-teamcity commented Sep 7, 2022

cockroach-teamcity commented Sep 8, 2022

cockroach-teamcity commented Sep 9, 2022

cockroach-teamcity commented Sep 10, 2022

cockroach-teamcity commented Sep 11, 2022

cockroach-teamcity commented Sep 12, 2022

cockroach-teamcity commented Sep 13, 2022

dt commented Sep 14, 2022

cockroach-teamcity commented Sep 2, 2022 •

edited by cockroach-jira-scripts

Loading

stevendanna commented Sep 7, 2022 •

edited

Loading