Is is possible for NCCL to add a retry mechanism when net flap happens

[ProHuper]

Currently, when Link Down error happens, the whole NCCL communication group will stop, and the whole training job will fail too, we have failover in job level, but restarting job takes a lot extra time.

For example:
In RoCE, when link down occurs, NCCL log shows this and fail:

In IB, when link down occurs, NCCL log shows this and fail:

In RoCE, dmesg shows:

In IB, UFM shows:

Link down won't last for a long time, usually after max to dozens of seconds, it will recover, during which network topo and other context information won't change. Can NCCL add a retry mechanism when there is a network flap?