Improving consistency between SessionServer and DataServer

[atellwu]

The background

The reasons for the inconsistency between SessionServerServer and DataServer：

• SessionServer sending clientOff is not absolutely reliable
• SessionServer disconnection DataServer over 30 seconds triggers SessionServer Off, causing DataServer to be erased incorrectly
• DataServer outage may cause datum loss in memory queues
• DataServer outage exceeds N (N is the number of copies), and some datum will be permanently missible.
• Disadvantage of consistency hash: When DataServer list changes frequently, SessionServers may send datum to incorrect DataServer at certain times, resulting in dirty datum and missing datum

How to ensure consistency?

• Consistency between client and SessionServer:
  • Pub / unpub sent by client: Write directly to SessionServer memory, which is 100% successful. Success means that client and SessionServer are identical.
  • Client disconnection: SessionServer is connected aware and executes clientOff to delete pub datum in memory. Because of the bolt heartbeat, the reliability of the perception of connection survival can be guaranteed.
• Eventual consistency between SessionServer and DataServer
  • SessionServer asynchronous threads are responsible for sending specific pub/unpub/clientOff datum to DataServer, retry if failed, and drop and log if retry fails a certain number of times.
    • Special scenario: If the client disconnects/SessionServer downtime, the client will reconnect to other SessionServers and send all datum again, and the old datum will be deleted because of the client off/SessionServer off/expired.
  • SessionServer regularly sends renew (including heartbeat and checksum) to ensure that the wrong datum can be corrected; Use expired mechanism to regularly clean up expired datum.

Design for SessionServer

Design for DataServer

[dzdx]

support by SlotDiffSyncer