dm dump task cannot backup to azure

[River2000i]

What did you do?

create dump task with azure dir by openapi

{
"task": {
        "name": "test-dump",
        "task_mode": "dump",
        "meta_schema": "dm-meta",
        "enhance_online_schema_change": true,
        "on_duplicate": "error",
        "target_config": {
            "host": "127.0.0.1",
            "port": 4000,
            "user": "root",
            "password": "",
        },
        "table_migrate_rule": [
            {
                "source": {
                    "source_name": "SOURCE1_NAME",
                    "schema": "openapi",
                    "table": "*",
                },
                "target": {"schema": "openapi", "table": "t"},
            }
        ],
        "source_config": {
            "full_migrate_conf": {
                "export_threads": 4,
                "import_threads": 16,
                "data_dir": "azure://external/dmtest4?account-name={account-name}&account-key={account-key}",
                "consistency": "auto",
            },
            "incr_migrate_conf": {"repl_threads": 16, "repl_batch": 100},
            "source_conf": [
                {"source_name": "SOURCE1_NAME"},
            ],
        }
    }
}

What did you expect to see?

[2024/12/19 21:32:27.443 +08:00] [INFO] [utils.go:145] ["remove all"] [url=azure://external/dmtest] [dir="azure://external/dmtest?account-name={account-name}&account-key={account-key}"]
[2024/12/19 21:32:27.551 +08:00] [INFO] [utils.go:148] ["walk dir"] [path=test-blob.txt]
[2024/12/19 21:32:27.755 +08:00] [ERROR] [dumpling.go:157] ["fail to remove output directory"] [task=test-dump] [unit=dump] [directory="azure://external/dmtest?account-name={account-name}&account-key={account-key}"] [error="Failed to delete azure blob file, file info: bucket(container)='external', key='dmtest': DELETE https://dmtestdump.blob.core.windows.net/external/dmtest\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 The specified blob does not exist.\nERROR CODE: BlobNotFound\n--------------------------------------------------------------------------------\n�<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.\n</Message></Error>\n--------------------------------------------------------------------------------\n"]
[2024/12/19 21:32:27.757 +08:00] [INFO] [subtask.go:340] ["unit process returned"] [subtask=test-dump] [unit=Dump] [stage=Paused] [status={}]
[2024/12/19 21:32:27.757 +08:00] [ERROR] [subtask.go:361] ["unit process error"] [subtask=test-dump] [unit=Dump] ["error information"="ErrCode:32001 ErrClass:\"dump-unit\" ErrScope:\"internal\" ErrLevel:\"high\" Message:\"mydumper/dumpling runs with error, with output (may empty): fail to remove output directory: azure://external/dmtest?account-name={account-name}&account-key={account-key}\" RawCause:\"Failed to delete azure blob file, file info: bucket(container)='external', key='dmtest': DELETE https://dmtestdump.blob.core.windows.net/external/dmtest\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 The specified blob does not exist.\\nERROR CODE: BlobNotFound\\n--------------------------------------------------------------------------------\\n\\357\\273\\277<?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.\\n</Message></Error>\\n--------------------------------------------------------------------------------\\n\" "]

What did you see instead?

delete dir failure

func RemoveAll(ctx context.Context, dir string, storage bstorage.ExternalStorage) error {
	var err error
	if storage == nil {
		storage, err = CreateStorage(ctx, dir)
		if err != nil {
			return err
		}
	}

	log.L().Info("remove all", zap.String("url", storage.URI()), zap.String("dir", dir))
	err = storage.WalkDir(ctx, &bstorage.WalkOption{}, func(filePath string, size int64) error {
		log.L().Info("walk dir", zap.String("path", filePath))
		err2 := storage.DeleteFile(ctx, filePath)
		if errors.Cause(err2) == gstorage.ErrObjectNotExist {
			// ignore not exist error when we delete files
			return nil
		}
		return err2
	})
	if err == nil {
		err = storage.DeleteFile(ctx, "")
		if errors.Cause(err) == gstorage.ErrObjectNotExist {
			// ignore not exist error when we delete files
			return nil
		}
	}
	return err
}

No response

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

(paste DM version here, and you must ensure versions of dmctl, DM-worker and DM-master are same)

Upstream MySQL/MariaDB server version:

(paste upstream MySQL/MariaDB server version here)

Downstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

How did you deploy DM: tiup or manually?

(leave TiUP or manually here)

Other interesting information (system version, hardware config, etc):

>
>

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)

[River2000i]

/assign

[River2000i]

/severity moderate