Consider adding Progress/Cancellation to reconnect loop

[markfields]

I'm considering whether hosts would benefit from progress/cancelation in the connectCore loop, e.g. when thinking about this bug (server outage caused JoinSession to return 500 for a few hours).

We discussed this a bit in #8411 (comment), and it seems to me better to let the host define the policy, leveraging any driver-specific knowledge it has coupled with the error(s) being repeatedly hit. See #9558 for thoughts about formalizing what driver-specific info a host could pull from an ODSP driver error (that the Loader layer wouldn't be able to leverage).

[vladsud]

I'd love to better understand programing model here. In other places (where there is clear request/response pattern in the form of API call) it's obvious how to provide cancelation and progress reporting.
With connection flow, most of the connection APIs (see https://github.com/microsoft/FluidFramework/pull/9439/files) are fire-and-forget. And even if they were not, there are implicit flows (connect on boot) where we do not return anything to consumers.
But consumers do want to have a way to know if 429 happens and connection is essentially not progressing.

500 cases seems like from slightly different bucket, but it has similar requirements - give ability to host to react to abnormal patterns (continuous reconnects, as an example). I think that's fine, but I'd probably not tight it together with progress / cancellation, we do not have to use same pattern here.

[markfields]

This issue applies to part of the overarching design discussed here: #9677

[vladsud]

@markfields, feel free to port to ADO, but we are closing pretty much all github issues.
If you do port, please answer my question above :)