roachtest: restore2TB/nodes=32 failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/109cf8705b773c0d3a1e7ab02ce63f764e101106

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=restore2TB/nodes=32 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=991714&tab=buildLog

The test failed on master:
	test.go:1037: test timed out (3h0m0s)
	test.go:639,cluster.go:1110,restore.go:244,cluster.go:1432,errgroup.go:58: /home/agent/work/.go/bin/roachprod run teamcity-991714-restore2tb-nodes-32:1 -- ./cockroach sql --insecure -e "
						RESTORE csv.bank FROM
						'gs://cockroach-fixtures/workload/bank/version=1.0.0,payload-bytes=10240,ranges=0,rows=65104166,seed=1/bank'
						WITH into_db = 'restore2tb'" returned:
		stderr:
		
		stdout:
		: signal: killed
	test.go:639,cluster.go:1453,restore.go:250: context canceled

[tbg]

This also had #31842.

[tbg]

Full of stuck Raft proposals.

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/bbc646fc6de90b59c0253fd682667715959fb657

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=restore2TB/nodes=32 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=993605&tab=buildLog

The test failed on master:
	test.go:1037: test timed out (3h0m0s)
	test.go:639,cluster.go:1110,restore.go:244,cluster.go:1432,errgroup.go:58: /home/agent/work/.go/bin/roachprod run teamcity-993605-restore2tb-nodes-32:1 -- ./cockroach sql --insecure -e "
						RESTORE csv.bank FROM
						'gs://cockroach-fixtures/workload/bank/version=1.0.0,payload-bytes=10240,ranges=0,rows=65104166,seed=1/bank'
						WITH into_db = 'restore2tb'" returned:
		stderr:
		
		stdout:
		: signal: killed
	test.go:639,cluster.go:1453,restore.go:250: context canceled

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/5b5738084b8cdc769d5e7973921de5cae4457380

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=restore2TB/nodes=32 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=995412&tab=buildLog

The test failed on master:
	test.go:1037: test timed out (3h0m0s)
	test.go:639,cluster.go:1118,restore.go:244,cluster.go:1440,errgroup.go:58: /home/agent/work/.go/bin/roachprod run teamcity-995412-restore2tb-nodes-32:1 -- ./cockroach sql --insecure -e "
						RESTORE csv.bank FROM
						'gs://cockroach-fixtures/workload/bank/version=1.0.0,payload-bytes=10240,ranges=0,rows=65104166,seed=1/bank'
						WITH into_db = 'restore2tb'" returned:
		stderr:
		
		stdout:
		: signal: killed
	test.go:639,cluster.go:1461,restore.go:250: context canceled

[tbg]

Hopefully fixed for good by #32594.