IIS In Process aspnetcore process failing to start under load

[adamzest]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When running our .Net 6 services under IIS in process 'sometimes' our services fail to start up properly, leaving the app pool running but the service completely unresponsive (in reality it looks like it didn't get as far as starting the dotnet application - see logs from aspnetcore). This tends to be following a reboot of a server or during an IISRESET where the CPU is very busy starting up many services/app pools, and a few fail to start the 'in process' process properly.

Having enabled aspnetcore debug tracing I can see a difference in output between success and failure of the service to startup. The key difference is that on failure we see this in the log:
Thread reading stdout/err hit timeout, forcibly closing thread.

Also note that if we switch the web.config to use 'outofprocess', then we are not able to recreate - i.e. the service always starts successfully.

aspnetcore startup failure.txt
aspnetcore startup success.txt

Based on the logs I believe this is happening before launching the service.

Expected Behavior

Service should reliably startup when using IIS in process hosting.

Steps To Reproduce

This is tricky... as per description this does not appear to be related to the application being hosted, but rather the aspnetcore hosting process.

Configure IIS server with many app pools (aspnet core, in process) that auto start.
Reboot server/IISRESET inducing high cpu load
Some processes dont start properly leaving the app pool running but no requests are serviced (application doesnt appear to have been started)

Exceptions (if any)

AspNetCoreModuleV2 module outputs to debug log:
Thread reading stdout/err hit timeout, forcibly closing thread.

.NET Version

6.0.300

Anything else?

No response

[ghost]

Thanks for contacting us.
We're moving this issue to the .NET 7 Planning milestone for future evaluation / consideration. Because it's not immediately obvious that this is a bug in our framework, we would like to keep this around to collect more feedback, which can later help us determine the impact of it. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[adamzest]

We have now recreated this with an unmodified service created from the VS 2022 "ASP.Net Core Web App" template, so I would argue it is obvious that this is a bug in the framework.

[HaoK]

Can you give a detailed set of repro steps using the template to demonstrate the problem?

[ghost]

Hi @adamzest. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[adamzest]

Reproducible on multiple servers using Azure VMs (Standard E2as v4 (2 vcpus, 16 GiB memory), however it is hard to recreate without the specific circumstances. Our IT team would be happy to help with additional diagnostics or demonstration.

Our IIS Configuration is:

• 6 application, with always running app pools configured for 'no managed code'. Application can be the .Net 6 'ASP.Net Core Web App template' running 'in process' (the more instances of these services the greater the chance of catching 1 failing)
• 44 applications, with always running app pools running .Net Framework 4.7.2 WCF servers (what seems critical is that there is enough load to create 100% cpu load for several minutes following an IISRESET)

Steps:
Either reboot VM, or perform an IISRESET
Observe high CPU load for around 5 minutes whilst services are starting - this is critical to the repro.
Attempt to load the services via http requests
Some of the .net 6 'In process' apps do not respond to http requests. Those services will:

• have a started up app pool which does not detect failure
• are using minimal memory (around 5mb)
• have not executed any app code (we do not believe AspNetCoreModuleV2 actually started the process at all)
• AspNetCoreModuleV2 module outputs the following to its debug log: "Thread reading stdout/err hit timeout, forcibly closing thread."

[adamzest]

Any update on this please? It continues to plague us daily.

[HaoK]

Its hard for us to make much progress without a way to reproduce the issue so we can debug/take a deeper look. Can you try to isolate a small repro app with a series of steps to demonstrate the issue?

[ghost]

Hi @adamzest. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.