roachtest: restoreTPCCInc/nodes=10 failed #85452

cockroach-teamcity · 2022-08-02T11:07:39Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on release-21.2 @ 31d61a4ca7cbc4df01e34a726a707dbe2eb6af91:

		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'" returned
		  | stderr:
		  | ERROR: importing 12872 ranges: Get "https://storage.googleapis.com/cockroach-fixtures/tpcc-incrementals/2021/05/21-020411.00/20210521/083000.00/660325993016360968.sst": stream error: stream ID 7; INTERNAL_ERROR; received from peer
		  | Failed running "sql"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach sql --insecure -e "
		  |   | 				RESTORE FROM '2021/05/21-020411.00' IN
		  |   | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  |   | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,restore.go:506,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:506
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #84162 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-master release-blocker]
roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!

Jira issue: CRDB-18299}

The text was updated successfully, but these errors were encountered:

adityamaru · 2022-08-03T14:59:14Z

@rhu713 assigning to you because stream error: stream ID 7; INTERNAL_ERROR; received from peer

cockroach-teamcity · 2022-08-05T09:56:18Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on release-21.2 @ ac6f3a94e8d00153220efb4528f17a27d231d27f:

		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'" returned
		  | stderr:
		  | ERROR: importing 12872 ranges: Get "https://storage.googleapis.com/cockroach-fixtures/tpcc-incrementals/2021/05/21-020411.00/660250420958527491.sst": stream error: stream ID 7; INTERNAL_ERROR; received from peer
		  | Failed running "sql"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach sql --insecure -e "
		  |   | 				RESTORE FROM '2021/05/21-020411.00' IN
		  |   | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  |   | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,restore.go:506,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:506
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-09T11:04:31Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on release-21.2 @ c2a7c3beee18554abd09a06d5c647b71da780827:

		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'" returned
		  | stderr:
		  | ERROR: importing 12872 ranges: Get "https://storage.googleapis.com/cockroach-fixtures/tpcc-incrementals/2021/05/21-020411.00/20210521/121500.00/660369963392925698.sst": stream error: stream ID 7; INTERNAL_ERROR; received from peer
		  | Failed running "sql"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach sql --insecure -e "
		  |   | 				RESTORE FROM '2021/05/21-020411.00' IN
		  |   | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  |   | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,restore.go:506,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:506
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-18T09:58:14Z

roachtest.restoreTPCCInc/nodes=10 failed with artifacts on release-21.2 @ c32478fc4b7eda315fd36569c131fba01ec05984:

		  | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'" returned
		  | stderr:
		  | ERROR: importing 12872 ranges: Get "https://storage.googleapis.com/cockroach-fixtures/tpcc-incrementals/2021/05/21-020411.00/20210521/063000.00/660302213847187462.sst": stream error: stream ID 7; INTERNAL_ERROR; received from peer
		  | Failed running "sql"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach sql --insecure -e "
		  |   | 				RESTORE FROM '2021/05/21-020411.00' IN
		  |   | 				'gs://cockroach-fixtures/tpcc-incrementals?AUTH=implicit'
		  |   | 				AS OF SYSTEM TIME '2021-05-21 14:40:22'"
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,restore.go:506,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:506
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: restoreTPCCInc/nodes=10 failed #79030 roachtest: restoreTPCCInc/nodes=10 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

msbutler · 2022-10-04T13:55:00Z

haven't seen this stream internal error in a while. closing this.

cockroach-teamcity added branch-release-21.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 2, 2022

cockroach-teamcity added this to the 21.2 milestone Aug 2, 2022

blathers-crl bot added the T-disaster-recovery label Aug 2, 2022

cockroach-teamcity mentioned this issue Aug 3, 2022

roachtest: restoreTPCCInc/nodes=10 failed #79030

Closed

adityamaru assigned rhu713 Aug 3, 2022

dt removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Aug 15, 2022

exalate-issue-sync bot assigned dt and unassigned rhu713 Aug 16, 2022

This was referenced Sep 2, 2022

roachtest: restoreTPCCInc/nodes=10 failed #87331

Closed

roachtest: restoreTPCCInc/nodes=10 failed #87602

Closed

msbutler closed this as completed Oct 4, 2022

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: restoreTPCCInc/nodes=10 failed #85452

roachtest: restoreTPCCInc/nodes=10 failed #85452

cockroach-teamcity commented Aug 2, 2022 •

edited by cockroach-jira-scripts

Loading

adityamaru commented Aug 3, 2022

cockroach-teamcity commented Aug 5, 2022

cockroach-teamcity commented Aug 9, 2022

cockroach-teamcity commented Aug 18, 2022

msbutler commented Oct 4, 2022

roachtest: restoreTPCCInc/nodes=10 failed #85452

roachtest: restoreTPCCInc/nodes=10 failed #85452

Comments

cockroach-teamcity commented Aug 2, 2022 • edited by cockroach-jira-scripts Loading

adityamaru commented Aug 3, 2022

cockroach-teamcity commented Aug 5, 2022

cockroach-teamcity commented Aug 9, 2022

cockroach-teamcity commented Aug 18, 2022

msbutler commented Oct 4, 2022

cockroach-teamcity commented Aug 2, 2022 •

edited by cockroach-jira-scripts

Loading