Improve the rate of thread injection for blocking due to sync-over-async

[kouvel]

• By default add threads more quickly when thread pool worker threads are blocking due to sync-over-async
• This would cover Task.Wait() and similar Task-based synchronous calls from thread pool threads. For now, it would not cover other kinds of blocking.
• Threads would be added quickly initially (without a delay for the first N additional threads) and progressive delays would be induced before adding each additional thread as the thread count increases (increased delay for each subsequent batch of M additional threads)
• Memory usage and limits would be used to determine when to stop adding threads quickly
• Heuristics would be configurable to enable controlling rates and limits

[joshudson]

I tried to code this a few years back, and discovered that the task scheduler itself is not a replaceable component. Nevertheless, here's logically what my never-executed code did:

[ThreadStatic] bool IsTaskSchedulerThread;

TaskAwaiter.GetResult() {
    if (!IsCompleted) {
        if (IsTaskSchedulerThread && !Scheduled) {
           // Absolutely guarantee there's enough threads
           try {
              StartTaskSchedulerThread();
           } catch {
              throw;
           }
        }
        // I honestly don't remember how this set up to receive the notification that the task finished. I think it involved Monitor.Enter/Exit to block the thread and wait for the other thread to run but there was a wrinkle to it
    }
    // Return result or throw exception
}

void StartTaskSchedulerThread()
{
    var th = new Thread(TaskSchedulerThread);
    th.IsBackgrounThread = true;
    th.Start();
}

void TaskSchedulerThread()
{
    var mccontrol = new AutoResetEvent(false);
    IsTaskSchedulerThread = true;
    for (;;) {
        while (TaskQueue.TryGetValue(Action continuation))
            continuation();
        PushWaitingThread(mccontrol);
        mccontrol.WaitOne(1000);
    }
}

Point being, if you tried to async over sync over async the code added another thread to the pool and threw back to the guy who called .GetResult() if it couldn't so it didn't deadlock.

I never had to figure out when to stop adding threads because I couldn't get it running. I had some idea to trace the chains and determine if there were still enough pool threads or not.

[kouvel]

@joshudson, I wonder why you needed to do this. I consider adding more threads to be only a workaround at best for the real problem, and there are fair reasons for running into that problem, and fair reasons for having a reasonable workaround such as this. I'm curious what prompted you to explore this and wonder if there might be a better solution.

[joshudson]

@kouvel: Two reasons. 1) Everybody was doing async over sync over async back then, including MVC Razr pages and FileStream itself. 2) I intended to catch exceptions escaping from the invocations of Task.ContinueWith() and passing them on to the next completion handler in line so that things like the still-extant bugs in FileStream where the IO completion callback can throw wouldn't bring the whole application down. It wouldn't have worked for Task.WaitAll() very well but it would have worked for async/await anything.

[kouvel]

things like the still-extant bugs in FileStream where the IO completion callback can throw wouldn't bring the whole application down

I'm not familiar with this issue, and there are some other aspects of your scenario that I'm not familiar with. @stephentoub?

[stephentoub]

so that things like the still-extant bugs in FileStream where the IO completion callback can throw wouldn't bring the whole application down

I'm not familiar with this issue

I don't know what this is referring to. What bugs?

[joshudson]

@stephentoub : You aren't the guy who has to take the call when the services collapsed rather than recover after everybody using the server tried a memory-intensive process at the same time. However I'd rather not debate this here. I provided some ideas from a path that didn't work out because I have some insight on the problem at hand.

[kouvel]

@joshudson, from your example anyway I believe this fix would be configurable enough to set a configuration to make it work similarly. By default some throttling would be done but it would still add threads much faster than currently.

[kouvel]

@joshudson, to clarify, this issue is specific to sync over async. You mentioned async over sync over async, this change is not intended to handle the async over sync part decently in a configurable way when the sync work is done on the thread pool, unless the sync part is waiting on a task. Could you describe in more detail how the async over sync over async arises?

[joshudson]

@kouvel: We tracked down and eliminated every last instance in the years since and won't deliberately write anymore because of the deadlocks. It used to arise due to mutually incompatible interfaces (mainly IEnumerable and IDisposable clashing with HttpClient but also Stream clashing with itself before IAsyncDisposable was a thing). In theory, IAsyncEnumerable can still clash with HttpClient but I doubt the case will arise in practice. "unless the sync part is waiting on a task" was indeed the case I was writing about.

[kouvel]

"unless the sync part is waiting on a task" was indeed the case I was writing about.

Ok then it sounds reasonable that a fix for this issue would be sufficient for your scenario (though perhaps with some config parameters to make it work the way you want)