[Blazor] Better diagnostics for Blazor server

[javiercn]

We get a lot of questions/issues opened about Blazor Server memory management. It is clear that this is a confusing aspect for customers and that they run into issues when they inadvertently introduce problematic patterns in their apps, like components rendering a large set of elements which can cause the memory to grow significantly.

When a customer finds itself in this situation, they generally open an issue with us that suggest there is a memory leak in the framework (which has not been the case so far) and we get into a discussion where we explain how Blazor deals with circuits, how memory works in Server GC and how to troubleshoot memory issues. The general points that people miss are:

• Circuits are alive for a given time after the session ends (up to a maximum number of circuits) to support reconnection.
• The server will only collect the circuits when it triggers a collection for the generation they live in, and in general, they will tend to go to Gen2.
• Server GC will only trigger a Gen2 collection when the server is running low on memory.

It will be extremely helpful if we can add additional diagnostics to help bring visibility into some of the internal aspects of how Blazor server operates. Specifically:

• dotnet-counters to track on a given server instance over time:
  • The number of connected circuits.
  • The number of disconnected circuits.

This makes it trivial to prove that we are handling the circuit lifetime correctly (or to spot when we are not) and since all the allocated framework memory is rooted by a circuit, it makes it easy to prove that the memory issue is not on the framework.

Further diagnostics to help identify problematic circuit instances using clrMD to identify circuits consuming a large amount of memory. We can do this as follows:

• Attach to the live process (or a memory dump).
• Walk the heap to find all the CircuitHost instances.
• For each CircuitHost instance, introspect its renderer and look at the ComponentState instances.
• For each ComponentState, we look at the backing field for the CurrentRenderTree
• On the RenderTreeBuilder we look at RenderTreeFrameArrayBuilder we look at the _items field to check the length of the array as well as the _itemsInUse.
• With this information, we can point out when a component rendered a large number of elements in the past or is currently rendering a large number of elements.
  • If it is currently rendering a large number of elements, _itemsInUse will be large.
  • If it has rendered a large number of elements in the past _items will be large.

With this data, we should be able to point out what components rendered a large number of elements currently or in the past as well as what circuit they belong(ed) to and its current state (whether the circuit is active, disconnected, or waiting to be collected) and it should also help discard the framework or the component rendering as the source of the memory issues.

Note that this could also be extended to use something similar to !objsize on a component instance to track how much memory that component is allocating, which in turn would help identify components in the app that are holding on to a lot of memory (think for example a component that has a large list in memory).

This last bit is not only useful for Blazor Server but it might be useful too for Blazor Webassembly and Desktop.

Another way to identify issues with the app this issue would be to check the ArrayPool<RenderTreeFrame> to identify large buffers that might have been allocated by the app in the past.

Examples of issues in this area in the past (there are likely more):
#43221
#39238
#39156
#33547
#33077
#32588
#30210
#21991
#18556
#17945
#14545
#10623

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[mkArtakMSFT]

Leaving this here for now as it may help customers who come across issues where they need to investigate Blazor Server related issues: https://github.com/javiercn/BlazorDumpAnalysisTool/tree/main/BlazorDumpAnalysis

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.