cancel import into task failed with error “Error 1105 (HY000): task manager is not initialized” and cancel task stuck when cancel again

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run import into
2、inject network partition between ddl owner and one of tikv
3、tidb oom
4、cancel import into task
cancel import into task failed with error “Error 1105 (HY000): task manager is not initialized”
5、cancel import into task again
cancel import into task stuck

2. What did you expect to see? (Required)

cancel import into task can success

3. What did you see instead (Required)

a、cancel import into task failed with error “Error 1105 (HY000): task manager is not initialized”
tidb log:
[2024/01/02 22:02:15.879 +08:00] [WARN] [manager.go:125] ["start manager failed"] [scope=] ["retry times"=10] [error="[planner:1054]Unknown column 'cpu_count' in 'field list'"]
[2024/01/02 22:02:20.887 +08:00] [WARN] [manager.go:125] ["start manager failed"] [scope=] ["retry times"=20] [error="[planner:1054]Unknown column 'cpu_count' in 'field list'"]
[2024/01/02 22:02:25.895 +08:00] [ERROR] [domain.go:1502] ["dist task executor manager start failed"] [error="[planner:1054]Unknown column 'cpu_count' in 'field list'"]
[2024/01/02 22:02:25.895 +08:00] [INFO] [wait_group_wrapper.go:140] ["background process exited"] [source=domain] [process=distTaskFrameworkLoop]
[2024/01/02 22:02:37.375 +08:00] [INFO] [tidb.go:286] ["rollbackTxn called due to ddl/autocommit failure"]
[2024/01/02 22:02:37.375 +08:00] [WARN] [session.go:2240] ["run statement failed"] [conn=3418357772] [session_alias=] [schemaVersion=88] [error="task manager is not initialized"] [session="{\n "currDBName": "",\n "id": 3418357772,\n "status": 2,\n "strictMode": true,\n "user": {\n "Username": "root",\n "Hostname": "10.233.109.59",\n "CurrentUser": false,\n "AuthUsername": "root",\n "AuthHostname": "%",\n "AuthPlugin": "mysql_native_password"\n }\n}"]
[2024/01/02 22:02:37.375 +08:00] [INFO] [conn.go:1132] ["command dispatched failed"] [conn=3418357772] [session_alias=] [connInfo="id:3418357772, addr:10.233.109.59:57346 status:10, collation:utf8mb4_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="cancel import job 120004;"] [txn_mode=PESSIMISTIC] [timestamp=0] [err="task manager is not initialized\ngithub.com/pingcap/tidb/pkg/disttask/framework/storage.GetTaskManager\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/storage/task_table.go:108\ngithub.com/pingcap/tidb/pkg/executor.(*ImportIntoActionExec).Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/import_into.go:259\ngithub.com/pingcap/tidb/pkg/executor/internal/exec.Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/internal/exec/executor.go:309\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:1245\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:990\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:816\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:597\ngithub.com/pingcap/tidb/pkg/session.runStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:2386\ngithub.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:2228\ngithub.com/pingcap/tidb/pkg/server.(*TiDBContext).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/driver_tidb.go:293\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1974\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleQuery\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1748\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).dispatch\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1322\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1101\ngithub.com/pingcap/tidb/pkg/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/server.go:701\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2024/01/02 22:02:45.424 +08:00] [INFO] [tidb.go:286] ["rollbackTxn called due to ddl/autocommit failure"]

b、cancel import into task again but cancel import into task stuck
tidb logs：
[2024/01/02 22:03:18.191 +08:00] [INFO] [task_executor.go:428] [rollback] [type=ImportInto] [task-id=120004] [step=write&ingest]
[2024/01/02 22:03:18.479 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/02 22:03:18.779 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/02 22:03:19.078 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/02 22:03:19.379 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/02 22:03:19.678 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/02 22:03:19.979 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
...
[2024/01/03 09:53:48.779 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:49.079 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:49.379 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:49.678 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:49.979 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:50.279 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:50.579 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:50.879 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:51.179 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:51.479 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:51.778 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]
[2024/01/03 09:53:52.079 +08:00] [INFO] [manager.go:265] [onCanceledTasks] [task-id=120004]

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v7.6.0-alpha
Edition: Community
Git Commit Hash: d6aeaf0
Git Branch: heads/refs/tags/v7.6.0-alpha
UTC Build Time: 2023-12-29 11:43:42
GoVersion: go1.21.5
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-01-02T22:02:36.952+0800

[Lily2025]

/assign D3Hunter

[D3Hunter]

[2024/01/02 22:02:15.879 +08:00] [WARN] [manager.go:125] ["start manager failed"] [scope=] ["retry times"=10] [error="[planner:1054]Unknown column 'cpu_count' in 'field list'"]

the error is caused by above msg after #49875 merged, the cluster cannot upgrade in-place in this case, we should recreate it or alter table manually.

close it