What's TiDB correct shutdown steps?

[DavisLinDB]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.

I have 7 VMs to set a group of TiDB.
3 PDs (PD1, PD2, PD3)
3 TiKVs (TiKV1, TiKV2, TiKV3)
1 TiDB

Each time I start the group by the order: PD1 > PD2 > PD3 >TiKV1 > TiKV2 > TiKV3 > TiDB.
When shutdown the group, the order is : TiDB > TiKV3 > TiKV2 > TiKV1 > PD3 > PD2 > PD1.
I direct use SUDO SHUTDOWN NOW in command mode to close VM and services.
When I restart the group next time, TiDB need to restart several times then start successful.

Could you give some suggestion?

2. What did you expect to see?
   TiDB starts correctly.

3. What did you see instead?
   2017/05/15 08:13:14 printer.go:31: [info] Welcome to TiDB.
   2017/05/15 08:13:14 printer.go:32: [info] Version:
   2017/05/15 08:13:14 printer.go:33: [info] Git Commit Hash: b2a7b87
   2017/05/15 08:13:14 printer.go:34: [info] UTC Build Time: 2017-05-13 04:52:43
   2017/05/15 08:13:14 client.go:96: [info] [pd] create pd client with endpoints [10.140.0.2:2379 10.140.0.3:2379 10.14
   0.0.4:2379]
   2017/05/15 08:13:14 client.go:186: [info] [pd] leader switches to: http://10.140.0.2:2379, previous:
   2017/05/15 08:13:14 client.go:114: [info] [pd] init cluster id 6415378353808530481
   2017/05/15 08:13:14 gc_worker.go:104: [info] [gc worker] 5702cac80a80002 start.
   2017/05/15 08:13:31 txn.go:169: [info] [kv] Rollback txn 391862286985723907
   2017/05/15 08:13:31 session.go:953: [fatal] check bootstrapped err [try again later]: backoffer.maxSleep 15000ms is
   exceeded, errors:
   send tikv request error: dial tcp 10.140.0.7:20160: getsockopt: connection refused, ctx: region_id:5001 region_epoch
   :<conf_ver:3 version:3 > peer:<id:5004 store_id:5 > , try next peer later
   stale_epoch:<>
   send tikv request error: dial tcp 10.140.0.5:20160: getsockopt: connection refused, ctx: region_id:5001 region_epoch
   :<conf_ver:3 version:3 > peer:<id:5002 store_id:1 > , try next peer later
   2017/05/15 08:13:36 printer.go:31: [info] Welcome to TiDB.
   2017/05/15 08:13:36 printer.go:32: [info] Version:
   2017/05/15 08:13:36 printer.go:33: [info] Git Commit Hash: b2a7b87
   2017/05/15 08:13:36 printer.go:34: [info] UTC Build Time: 2017-05-13 04:52:43
   2017/05/15 08:13:36 client.go:96: [info] [pd] create pd client with endpoints [10.140.0.2:2379 10.140.0.3:2379 10.14
   0.0.4:2379]
   2017/05/15 08:13:36 client.go:186: [info] [pd] leader switches to: http://10.140.0.2:2379, previous:
   2017/05/15 08:13:36 client.go:114: [info] [pd] init cluster id 6415378353808530481
   2017/05/15 08:13:36 gc_worker.go:104: [info] [gc worker] 5702cadd0980002 start.
   2017/05/15 08:13:51 txn.go:169: [info] [kv] Rollback txn 391862292621819907
   2017/05/15 08:13:51 session.go:953: [fatal] check bootstrapped err [try again later]: backoffer.maxSleep 15000ms is
   exceeded, errors:
   send tikv request error: dial tcp 10.140.0.6:20160: getsockopt: connection refused, ctx: region_id:5001 region_epoch
   :<conf_ver:3 version:3 > peer:<id:5003 store_id:4 > , try next peer later
   send tikv request error: dial tcp 10.140.0.7:20160: getsockopt: connection refused, ctx: region_id:5001 region_epoch
   :<conf_ver:3 version:3 > peer:<id:5004 store_id:5 > , try next peer later
   stale_epoch:<>

4. What version of TiDB are you using (tidb-server -V)?
   2017/05/15 08:13:36 printer.go:32: [info] Version:
   2017/05/15 08:13:36 printer.go:33: [info] Git Commit Hash: b2a7b87
   2017/05/15 08:13:36 printer.go:34: [info] UTC Build Time: 2017-05-13 04:52:43

[shenli]

@DavisLinDB After starting pd-server and tikv-server, you should sleep for a while. pd-server/tikv-server need a few seconds (it depends on how many data is there and how fast is your disk/cpu.) to load data. You could write a script like:

start pd-server command
sleep 3
start tikv-server command
sleep 3
start tidb-server command

[morgo]

A related request is for tidb-server to support the SHUTDOWN command:

tidb> shutdown;
ERROR 1105 (HY000): line 1 column 8 near "" (total length 8)

See #5046

[wwar]

This has been improved quite a bit. On startup: the components can start out of order and then wait for pd to be available. On shutdown, the tidb-server supports the SHUTDOWN command.

I believe this issue can be closed.