Move async user tasks to thread

[fjetter]

This came up in #7320 in context of cancellation of these tasks during worker closing

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       20 files  ±0         20 suites  ±0   11h 28m 28s ⏱️ - 30m 24s
  3 754 tests ±0    3 641 ✔️  -   2     106 💤 ±0    7 ❌ +  2 
36 314 runs  ±0  34 532 ✔️  - 29  1 748 💤 ±0  34 ❌ +29 

For more details on these failures, see this check.

Results for commit 2241969. ± Comparison against base commit 9255987.

♻️ This comment has been updated with latest results.

[wence-]

Could now also remove the definition of (the now unused) apply_function_async

[hendrikmakait]

LGTM. Thanks, @fjetter!

[gjoseph92]

+1 on this approach, I'd love to not let user code run in our event loop.

This also would have made #5997 a bit easier, since we'd lose the ability to cancel user async tasks (so we wouldn't have to special-case rescheduling them).

[fjetter]

FYI this is currently soft blocked. Prefect is using this feature quite heavily and they observed some change in behavior relating to exception handling.
Since they are a big user of this feature and we're not in a rush, we'll wait a bit for them to look into it

[zanieb]

@fjetter — just to confirm, this means that user-submitted async tasks are now running on multiple threads and event loops instead of a single thread and event loop?

[fjetter]

just to confirm, this means that user-submitted async tasks are now running on multiple threads and event loops instead of a single thread and event loop?

Yes, exactly.

[zanieb]

This should be good to go from our end! Thanks for letting us investigate.