DM secondsBehindMaster of query-status not accurate

[lichunzhu]

What did you do?

Deploy and start a dm cluster and then start a new dm task.
Execute some ddls on the existing tables.

What did you expect to see?

secondsBehindMaster of query-status can get updated from time to time.

What did you see instead?

secondsBehindMaster of query-status keeps increasing.

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

Release Version: v5.4.0-master
Git Commit Hash: 43a1d78f94db503ef991919ea9a6f4609c07bf11
Git Branch: master
UTC Build Time: 2022-01-25 03:43:31
Go Version: go version go1.16.4 darwin/amd64

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)

[lichunzhu]

Root Cause

ddlJobIdx bucket is never reset.

https://github.com/pingcap/tiflow/blob/release-5.3/dm/syncer/syncer.go#L1237

https://github.com/pingcap/tiflow/blob/master/dm/syncer/syncer.go#L1374

[lance6716]

another request is updating the secondsBehindMaster even DM block. Don't need to be too precise, just make it bigger and bigger
/cc @Ehco1996 Should I re-open #3695 so you can fix it in separate PR?

[Ehco1996]

another request is updating the secondsBehindMaster even DM block. Don't need to be too precise, just make it bigger and bigger /cc @Ehco1996 Should I re-open #3695 so you can fix it in separate PR?

got this