Add inline scheduler option for Sockets transport

[BrennanConroy]

Fixes #20952

Sockets Default + Kestrel Default : 1,187,598 RPS
Sockets Inline + Kestrel Inline: 1,242,924 RPS
Sockets Inline + Kestrel Default: 1,072,542 RPS
Sockets Default + Kestrel Inline: 1,153,500 RPS

[davidfowl]

RunOnTransportThread

[BrennanConroy]

Doesn't sound unsafe enough!

[halter73]

UnsafeRunOnTransportThread!

[pranavkm]

Suggested change

	public bool UnsafeInlineScheduling { get; set; }
	public bool UnsafePreferInlineScheduling { get; set; }

[pranavkm]

• We add the API and get the API added in dotnet/runtime. Setting our API should set both.
• We add the API and We use private reflection to set it on the socket itself. Setting our API should set both.
• We don't expose a public API and we also consume the env variable. (and we have an opt-out for weird cases)

[BrennanConroy]

Looks like the env variable in Runtime globally affects some settings and even if you use private reflection to set the per Socket setting you don't affect the other things the env variable does. This results in massive perf regressions, i.e. 100k RPS without the env variable but with private reflection vs. 1.2M RPS with the env variable and no private reflection.

https://github.com/tmds/runtime/blob/6ea1256bf8bd9ce6c1db278603b332cec144ba75/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs#L26

[ghost]

Thank you for submitting this for API review. This will be reviewed by @dotnet/aspnet-api-review at the next meeting of the ASP.NET Core API Review group. Please ensure you take a look at the API review process documentation and ensure that:

• The PR contains changes to the reference-assembly that describe the API change. Or, you have included a snippet of reference-assembly-style code that illustrates the API change.
• The PR describes the impact to users, both positive (useful new APIs) and negative (breaking changes).
• Someone is assigned to "champion" this change in the meeting, and they understand the impact and design of the change.

[pranavkm]

Explain how this can hurt your performance. Also document the DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS environment variable.

[BrennanConroy]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[adamsitnik]

It's great that this feature made it to 5.0! 👍