Improve usability in cdc client

[amyangfei]

Feature Request

Is your feature request related to a problem? Please describe:

Currently we use both cdc cli and HTTP API to manage changefeed, some features are not easy to use.

Describe the feature you'd like:

• 1. Providing both cdc cli and HTTP API is not user friendly, maybe we can move the HTTP API providing pause/resume/stop of changefeed into cdc cli
• 2. cdc cli capture list doesn't include the binding status_ip:status_port of capture
• 3. when a changefeed is deleted using curl -X POST -d "admin-job=3&cf-id=xxx", cdc cli changefeed list doesn't include this changefeed, bug cdc cli changefeed query --changefeed-id=xxx can query it

➜  ./cdc cli changefeed list
[]

➜  ./cdc cli changefeed query --changefeed-id=3f4c9771-46fd-4ed7-a6e5-31247d74f7bf
{
        "info": null,
        "status": {
                "resolved-ts": 416523083566022657,
                "checkpoint-ts": 416523083304402945,
                "admin-job-type": 3
        },
        "count": 0,
        "task-status": []
}

• 4. when a changefeed is deleted using curl -X POST -d "admin-job=3&cf-id=xxx", resume this changefeed has weird output (127.0.0.1:8303 is owner) ref: write ut cases for operator package ticdc#541

➜  curl -X POST -d "admin-job=2&cf-id=3f4c9771-46fd-4ed7-a6e5-31247d74f7bf" http://127.0.0.1:8303/capture/owner/admin
{
 "status": true,
 "message": ""
}%
➜  curl -X POST -d "admin-job=2&cf-id=3f4c9771-46fd-4ed7-a6e5-31247d74f7bf" http://127.0.0.1:8302/capture/owner/admin
curl: (52) Empty reply from server

• 5. a paused changefeed can't be removed Refine some state transition when handling admin job #705

➜  curl -X POST -d "admin-job=1&cf-id=8e512859-881f-4016-ac22-f5c1fe51f965" http://127.0.0.1:8301/capture/owner/admin
{
 "status": true,
 "message": ""
}%

➜  etcdctl get /tidb/cdc --prefix
/tidb/cdc/capture/cf408eec-c22f-47de-927d-3055a62287f0
{"id":"cf408eec-c22f-47de-927d-3055a62287f0"}
/tidb/cdc/changefeed/info/8e512859-881f-4016-ac22-f5c1fe51f965
{"sink-uri":"mysql://[email protected]:3306/","opts":{},"create-time":"2020-05-09T11:32:59.540343728+08:00","start-ts":416545552339828737,"target-ts":0,"admin-job-type":1,"sort-engine":"memory","sort-dir":".","config":{"ddl-white-list":null,"filter-case-sensitive":false,"filter-rules":null,"ignore-txn-commit-ts":null,"sink-dispatch-rules":null,"mounter-worker-num":0}}
/tidb/cdc/job/8e512859-881f-4016-ac22-f5c1fe51f965
{"resolved-ts":416545555145031683,"checkpoint-ts":416545554882887683,"admin-job-type":1}
/tidb/cdc/owner/6a6c71cfa12e7151
cf408eec-c22f-47de-927d-3055a62287f0

➜  curl -X POST -d "admin-job=3&cf-id=8e512859-881f-4016-ac22-f5c1fe51f965" http://127.0.0.1:8301/capture/owner/admin
changefeed [8e512859-881f-4016-ac22-f5c1fe51f965] not found%

• 6. Client should be able to identify connection/auth issues when creating a changefeed.
• 7. Client should use a timeout instead of blocking forever when connecting a bad address.
• 8. cdc cli logs to file by default now, which is not suitable for a command line tool, better to log some critical information to stdout.
• 9. Add PD and TiKV version compatibility check before cdc server start up.
• 10. Some optimizations for cdc cli changefeed remove. Reduce object count and memory usage for TableInfo ticdc#564
• 11. Add a simple command to let user understand the state and failure conditions of the cdc server and changefeed. changefeed list or changefeed query -s have supported.
• 12. Client should show more detailed information in changefeed list it: Add basic failover it test ticdc#721
• 13. cdc cli meta should support clean up GC-TTL in PD. cmd: add a command to delete CDC GC TTL #652
• 14. Add a command to create mark tables for cyclic replication. cyclic: create mark tables if cyclic is enabled #611
• 15. Refine some output fields in cdc cli. puller: fix resolved ts lost ticdc#677

[overvenus]

I will try to implement the feature 1.

[overvenus]

Turns out, it would be better to implement feature 1 and feature 2 together, as 1 can use address of captures to pause/resume/stop changefeeds.

[overvenus]

I'm going to resolve 6 and 7.

[kolbe]

I think it's OK to provide both an HTTP API and a command-line tool, but the command-line tool should simply send calls to the API.

I think using the "admin-job" flag to manage resources is very confusing. An HTTP API should use HTTP verbs to manage resources rather than opaque and confusing integers.

[amyangfei]

All tasks completed