Switch drain_all call to be called from new thread instead of event loop.

[jeanluciano]

Moves the drain_all call to a new thread instead of the event loop.

closes: #16115

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[codspeed-hq]

CodSpeed Performance Report

Merging #16739 will not alter performance

_{Comparing jean/oss-5963-deadlock-when-deloying-flow-with-dependencies-that (ca4b321) with main (5b5043c)}

Summary

✅ 2 untouched benchmarks

[zzstoatzz]

hmm i think it might be good to add a unit or integration test for this one! i remember some weird shutdown / deadlock here

[zzstoatzz]

I still seem to remember some issue about spawning a new thread at interpreter shutdown a la python/cpython#116677 but maybe this has since been fixed and released in a recent version of python 3.12 cc @jlowin (I remember we both ran into this)

anyways if we're passing tests and not seeing any issues, I think this makes sense

[zzstoatzz]

comment should probably be updated to match new behavior

Suggested change

	# coroutine so we just schedule the drain in the global loop thread and continue
	# coroutine so we just schedule the drain in a new thread and continue

[cicdw]

the bug ticket says the bug is triggered with incremental=True?

[jeanluciano]

That was my mistake, it is incremental=False as that is what makes configure call _clearExistingHandlers