Main/backup: changing switching rules

[maxsharabayko]

Changing main backup switching logic (WIP).

Changes

• Member sockets being in a Fresh-Activated state are not used to silence other active links until they become stable. Fixes Do not silence parallel links if there are no stable links #1793.
• sendBackup_RetryWaitBlocked was not waiting for active links, but was waiting to activate backup links. It conflicts with the logic of the sending function, as all backup activation is handled previously in sendBackup_TryActivateIdleLink. Now waits for active links. Concern: if it happens (should not happen) that there is one idle (stand by) link in the group that was not activated previously, it will continuously report read-readiness, resulting in a CPU-intensive write-readiness checking loop.
• Moved some logic into a newly created sendBackup_SendOverActive function.
• Unstable link may become stable again if it continuously gets a response from the receiver (not on a single response).
• An unstable link that remains unstable for quite some time can be closed if there is at least one stable link in a group (?).
• If a backup link was activated and qualified as stable, and it has the same or a higher weight as the main link (now unstable), silence the unstable link without closing it (perform a switch).

[lgtm-com]

This pull request introduces 1 alert when merging 2bdb324 into a1bcd4a - view on LGTM.com

new alerts:

• 1 for Declaration hides parameter

[maxsharabayko]

Additional metrics checked

• Sender packet drops – not very persistent and receiver is likely to drop first.
• Sender-side pkt loss (based on NAK reports) percentage. Also varying. Hard to tell which level to use.
• Packet drop reported by the receiver in extended ACK (PR [core] WIP: Extending ACK with RCV drop total #1889).
  • Probing for stability for n x latency.
  • Easy to implement.
  • Can’t tell which packets were dropped – no drop stats for the whole group.

Experiment

• RTT 100ms, SRT latency 300ms.
• 30% loss on path weight 10.
• 4xlatency for probing period (to transition from unstable -> stable).
• Closing a member socket which remains unstable for SRTO_PEERIDLETMO
• New receiver drop total in ACK -> qualify as unstable.

Problem

• Breaking an unstable link from the group does not invoke a connection callback. This logic needs to be somehow implemented, and it is hitting the group-member synchronization difficulties (mutex lock order).

[maxsharabayko]

Member ordering rule by priority

1. Higher weight (highest priority)
2. By backup state (if equal weight):
   1. ACTIVE_STABLE
   2. ACTIVE_FRESH
   3. ACTIVE_UNSTABLE_WARY
   4. ACTIVE_UNSTABLE
   5. BROKEN
   6. STANDBY (IDLE)
   7. PENDING
3. By Socket ID (if equal weight and state): lower value first.