sql: flake in TestExperimentalRelocateNonVoters and TestRelocateNonVoters

[knz]

Describe the problem

Also: #96343

Found here: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelExtendedCi/8305842?showRootCauses=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

See error details below. Maybe this problem exists because of the random insertion of a test tenant?

Error Trace:  /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4308/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sql_test_/sql_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/multitenant_admin_function_test.go:88
                              /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4308/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sql_test_/sql_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/multitenant_admin_function_test.go:335
                              /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4308/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sql_test_/sql_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/multitenant_admin_function_test.go:878
                              /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4308/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sql_test_/sql_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/multitenant_admin_function_test.go:309
                              /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4308/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sql_test_/sql_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/multitenant_admin_function_test.go:850
          Error:        "\xf0" does not contain "\xbd"
          Test:         TestExperimentalRelocateNonVoters/ALTER_TABLE_x_EXPERIMENTAL_RELOCATE_NONVOTERS
          Messages:     tenant=system query=`ALTER TABLE t EXPERIMENTAL_RELOCATE NONVOTERS VALUES (ARRAY[2], 1);` leaseholder=3 replicas=[1 3 4 5] voting_replicas=[1 3 5] non_voting_replicas=[4] fromReplica=4 toReplica=2 row=0 col=0

    multitenant_admin_function_test.go:88:
          Error Trace:  /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4308/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sql_test_/sql_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/multitenant_admin_function_test.go:88
                              /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4308/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sql_test_/sql_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/multitenant_admin_function_test.go:335
                              /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4308/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sql_test_/sql_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/multitenant_admin_function_test.go:813
                              /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4308/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sql_test_/sql_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/multitenant_admin_function_test.go:310
                              /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4308/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sql_test_/sql_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/multitenant_admin_function_test.go:782
          Error:        "75" does not contain "55"
          Test:         TestRelocateNonVoters/ALTER_RANGE_x_RELOCATE_NONVOTERS
          Messages:     tenant=secondary query=`ALTER RANGE (SELECT min(range_id) FROM [SHOW RANGES FROM TABLE t]) RELOCATE NONVOTERS FROM 3 TO 1;` leaseholder=2 replicas=[2 3 4 5] voting_replicas=[2 4 5] non_voting_replicas=[3] fromReplica=3 toReplica=1 row=0 col=0

Jira issue: CRDB-23398

Epic CRDB-23344

[mgartner]

Flaked here too: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/8310118?showRootCauses=true&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

[adityamaru]

Flaked here too - https://teamcity.cockroachdb.com/viewLog.html?buildId=8564093&buildTypeId=Cockroach_UnitTests_BazelUnitTests

@ecwall can you take a look at this, please? Or we can skip the test while you investigate.

[blathers-crl]

Hi @ajstorm, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[ecwall]

I think this is caused by this, but I will need help from KV:

// TODO(aayush): If the AdminChangeReplicas call failed because it found that
// the range was already in the process of being rebalanced, we currently fail
// the statement. We should consider instead force-removing these learners
// when AdminChangeReplicas calls are issued by SQL.

cockroach/pkg/sql/relocate_range.go

Line 157 in e4924e2

// TODO(aayush): If the `AdminChangeReplicas` call failed because it found that

[ecwall]

Fixing the rangeIDs not matching by ignoring them.

The other error that occurs randomly is #95252 (comment).

I think this one is related to the comment by aayush.