Adding TiFlash node to a cluster with 20k tables is slow

[JaySon-Huang]

If we add a TiFlash node to a TiDB cluster that already has 20k tables, TiFlash may be slow to start.
We will DO Full Schema sync when starting the new TiFlash node. And no matter those tables have TiFlash replica or not, we will create empty tables on the TiFlash node.

During this period, the TiFlash node

• does not register itself as a store on PD
• does not start reporting metrics
• does not listen on the coprocessor port

The period could be quite long, especially when the user enables the feature "Encryption at Rest" (in DBaaS, for example), it takes more time for TiFlash to "be ready" on PD.

Actually, we met a situation that adding a TiFlash node (v4.0.9) to a cluster with more than 20k tables. It took about 15~20 minutes to become "normal" on the DBaaS dashboard. The user is confused that why TiFlash is "Unavailable" for 15 minutes after he/she create the node. (DBaaS show "unavailable" if a TiFlash is UP but not found in PD store list)

[JaySon-Huang]

We should break down this problem in different aspect:

1. Can we shorten the period of showing "unavailable" -> "normal"?
   In this aspect, we should think about how to make users see adding a TiFlash node with "no surprise". Even if the TiFlash node can not serve requests when it is initiating itself.
2. Is full schema sync a necessary step before serving as a "RaftStore" and serving as a "Coprocessor" and reporting "metrics"?
   a. If we register as a store in PD before full schema sync. I think there will be an issue: PD starts to add region peer on TiFlash store, then TiFlash will trigger schema sync while applying Raft Logs. If schema sync takes a long time, this will block all the Raft threads. Will this make some "surprise" for TiFlash/TiKV/PD?
3. Can we shorten the period of schema sync?
   a. Can we do not create tables with no TiFlash replica?
   b. Can we create fewer files when creating an empty table? (An empty table will create 17 files at least in v4.0.9)
   c. Can we create fewer encrypt meta when creating an empty table?

[JaySon-Huang]

cc @zanmato1984

[flowbehappy]

We should break down this problem in different aspect:

1. Can we shorten the period of showing "unavailable" -> "normal"?
   In this aspect, we should think about how to make users see adding a TiFlash node with "no surprise". Even if the TiFlash node can not serve requests when it is initiating itself.
2. Is full schema sync a necessary step before serving as a "RaftStore" and serving as a "Coprocessor" and reporting "metrics"?

I would rather choose to keep the users waiting, instead of using an "unhealthy" service. But we can definitely shorten the waiting time.

a. If we register as a store in PD before full schema sync. I think there will be an issue: PD starts to add region peer on TiFlash store, then TiFlash will trigger schema sync while applying Raft Logs. If schema sync takes a long time, this will block all the Raft threads. Will this make some "surprise" for TiFlash/TiKV/PD?

Can we only sync the specific table schema, instead of ALL schemas, under this scenario?

3. Can we shorten the period of schema sync?
   a. Can we do not create tables with no TiFlash replica?

Definitely, we should not create tables with no TiFlash replica. I can't see any reason for blocking this issue.

b. Can we create fewer files when creating an empty table? (An empty table will create 17 files at least in v4.0.9)
There are several improvements for this issue IMO:

• Avoid creating a real IStorage instance, but a fake one. Only materialize it before any data coming in.
• Only create files for PageStorage and DTFile before the first write request. And only create the first Segment before writing the first rows in DeltaMergeStore
• Only open more writable files in PageStorage when needed.

c. Can we create fewer encrypt meta when creating an empty table?

After the "log style improvement" for encryption-at-rest, I think this issue considered fixed. Besides, we are going to use EBS's encryption by default in the future.

[JaySon-Huang]

It took about 20 minutes to add a new TiFlash node to a cluster with 20k tables after "log style improvement for encryption-at-rest" enabled.
If we don't pick that fix, it will take hours or tens of hours 😓😓 @flowbehappy

[flowbehappy]

It took 20 minutes after "log style improvement for encryption-at-rest" enable.

And by calculation, an empty table needs ~60ms (20.0 * 60 * 1000 / 20000) to create. Which I don't think is a surprise by the current design. But we still need to improve though.

[JaySon-Huang]

In a QA test, adding a TiFlash node to a cluster with 80k tables, with the feature "Encryption at Rest" disabled. It took 20 min for TiFlash to be ready. Creating an empty table is about 15ms under this circumstance.

[JinheLin]

Design doc: https://docs.google.com/document/d/10EfjRugZ5SFVyMb2MC0DnovT2TtZ_LRR7W4HxqMWb54/edit?usp=sharing