import large zstd files may encounter error

[lance6716]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

generate a large zstd data file and import by

bin/tidb-lightning -d /tmp/test-zstd -backend tidb --tidb-port 4000

2. What did you expect to see? (Required)

no error

3. What did you see instead (Required)

many kind of errors, like

encode kv error in file coincheck.exchange_order_applies.000000000.sql.zst:0 at offset 42598379: tableLog too large

encode kv error in file coincheck.exchange_order_applies.000000000.sql.zst:0 at offset 52101109: tableLog too large

encode kv error in file coincheck.exchange_order_applies.000000000.sql.zst:0 at offset 11796469: invalid input: reserved block type encountered

4. What is your TiDB version? (Required)

[lance6716]

Seems it's caused by some race in zstd decoder, but go's race detector does not report any problem. And I set concurrency = 1 the problem is gone.

Will set concurrency = 1 as a workaround and report the problem to upstream

[lance6716]

maybe related to #53655