Unexpected values reported by GC.GetGCMemoryInfo()

[antonfirsov]

Context

Working on an attempt to improve array pooling strategies in ImageSharp. I wanted to use GCMemoryInfo.MemoryLoadBytes in a similar manner as done by TlsOverPerCoreLockedStacksArrayPool

Issue

I have a small test app doing some stressing work by allocating and retaing arrays of 1MB, and querying GC.GetGCMemoryInfo which seems to report weird values:
https://gist.github.com/antonfirsov/01bb6b7b8569027cd71988ff33c66228

My test environment: Windows 10, 64bit physical memory, i9 with 10 physical cores

A couple of unexpected things about GCMemoryInfo.MemoryLoadBytes:

• .NET Core 3.1 + 32bit seems to be the only configuration, where it works as I would expect: MemoryLoadBytes is gradually scaling with my actual memory usage.
• .NET 5.0/6.0 + 32bit: GCMemoryInfo.MemoryLoadBytes reports values 10-50 times higher than the actual memory usage. These values do not make sense in a 32 bit process.
• 64 bit (3.1, 5.0, 6.0): after allocating the first array, GCMemoryInfo.MemoryLoadBytes immediately jumps to a constant ~21GB value which doesn't change even if GC.GetTotalMemory() passes that threshold.

TotalAvailableMemoryBytes and HighMemoryLoadThreshold are also weird:

• 3.1 + 32 bit: again, this is the only configuration that seems to report expected values: 2GB / 1.8 GB
• 5.0/6.0 32 bit: 64 GB / 59GB
• 64bit all versions: it's reported to 64 GB / 59 GB, but my test app is fine passing the 64GB threshold without an OOM, which makes me wonder about the semantics of TotalAvailableMemoryBytes. Is it only reporting the amount of physical memory on my PC without any impact on GC behavior?

Note that if the MemoryLoadBytes issue is a bug, it must be breaking the pressure-triggered trimming of ArrayPool<T>.Shared.

/cc @Maoni0

[ghost]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Context

Working on an attempt to improve array pooling strategies in ImageSharp. I want to use GCMemoryInfo.MemoryLoadBytes in a similar manner as done by TlsOverPerCoreLockedStacksArrayPool

Issue

I have a small test app doing some stressing work by allocating and retaing arrays of 1MB, and querying GC.GetGCMemoryInfo which seems to report weird values:
https://gist.github.com/antonfirsov/01bb6b7b8569027cd71988ff33c66228

My test environment: Windows 10, 64bit physical memory, i9 with 10 physical cores

A couple of unexpected things about GCMemoryInfo.MemoryLoadBytes:

• .NET Core 3.1 + 32bit seems to be the only configuration, where it works as I would expect: MemoryLoadBytes is gradually scaling with my actual memory usage.
• .NET 5.0/6.0 + 32bit: GCMemoryInfo.MemoryLoadBytes reports values 10-50 times higher than the actual memory usage
• 64 bit (3.1, 5.0, 6.0): after allocating the first array, GCMemoryInfo.MemoryLoadBytes immediately jumps to a constant ~21GB value which doesn't change even if GC.GetTotalMemory(false) passes that threshold.

TotalAvailableMemoryBytes and HighMemoryLoadThreshold are also weird:

• 3.1 + 32 bit: again, this is the only configuration that seems to report expected values: 2GB / 1.8 GB
• 5.0/6.0 32 bit: 64 GB / 59GB
• 64bit all versions: it's reported to 64 GB / 59 GB, but my test app is fine passing the 64GB threshold without an OOM. What is the purpose of TotalAvailableMemoryBytes, if it's only reporting the amount of physical memory on my PC without any impact on GC behavior?

If MemoryLoadBytes has a bug, it should heavily impact the ability of pressure-triggered trimming of ArrayPool<T>.Shared.

/cc @Maoni0

Author:	antonfirsov
Assignees:	-
Labels:	area-GC-coreclr
Milestone:	-

[Sergio0694]

I've run the repro code on my machine, with this setup:

• Ryzen 2700X, so 8C/16T
• 16GB of RAM
• Targeting .NET 6
• Using the 6.0.100-preview.7.21351.6 SDK

This seems to confirm the issue on 64 bit systems. MemoryLoad gets pinned to a value around the 40% of the total memory, and TotalAvailableMemory doesn't seem related to the actual memory available for the GC. Preview and full log below.

Log (click to expand):

Is64BitProcess: True
Retained: 0 MB, GC Total: 1 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 64 MB, GC Total: 65 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 128 MB, GC Total: 129 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 192 MB, GC Total: 193 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 256 MB, GC Total: 257 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 320 MB, GC Total: 321 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 384 MB, GC Total: 385 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 448 MB, GC Total: 449 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 512 MB, GC Total: 513 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 576 MB, GC Total: 577 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 640 MB, GC Total: 641 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 704 MB, GC Total: 705 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 768 MB, GC Total: 769 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 832 MB, GC Total: 833 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 896 MB, GC Total: 897 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 960 MB, GC Total: 961 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1024 MB, GC Total: 1025 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1088 MB, GC Total: 1089 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1152 MB, GC Total: 1153 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1216 MB, GC Total: 1217 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1280 MB, GC Total: 1281 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1344 MB, GC Total: 1345 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1408 MB, GC Total: 1409 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1472 MB, GC Total: 1473 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1536 MB, GC Total: 1537 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1600 MB, GC Total: 1601 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1664 MB, GC Total: 1665 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1728 MB, GC Total: 1729 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1792 MB, GC Total: 1793 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1856 MB, GC Total: 1857 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1920 MB, GC Total: 1921 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 1984 MB, GC Total: 1985 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 2048 MB, GC Total: 2049 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 2112 MB, GC Total: 2113 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 2176 MB, GC Total: 2177 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
Retained: 2240 MB, GC Total: 2241 MB, TotalAvailableMemory:16300 MB, HighMemoryLoadThreshold:14670 MB, MemoryLoad:7824 MB
[...]

Full log: memoryinfo.log

Hope this helps! 😄

[Maoni0]

memory load means what's loaded in the physical memory. the test allocates ~1 mb object each time but it doesn't touch most of it so the memory load basically doesn't change. @gewarren could we please clarify this in the doc?.

if you actually touch the pages of the object, eg, via the TouchPage method in this benchmark, you'll see the MemoryLoad field go up.

as for TotalAvailableMemoryBytes and HighMemoryLoadThreshold values for "5.0/6.0 32 bit", are you sure this is a 32-bit app? GC just gets the total physical memory from the OS layer.

[antonfirsov]

if you actually touch the pages of the object, [..] you'll see the MemoryLoad field go up.

Thanks! I updated my test app to touch the pages, and the 64bit output makes sense on my 16GB laptop now:
https://gist.github.com/antonfirsov/01bb6b7b8569027cd71988ff33c66228#file-updated-output-64bit-txt

as for TotalAvailableMemoryBytes and HighMemoryLoadThreshold values for "5.0/6.0 32 bit", are you sure this is a 32-bit app?

Yes, I print the value of Environment.Is64BitProcess at the very beginning.

Here is the (TouchPage-updated) output for dotnet run -c Release -f netcoreapp3.1 --runtime win-x86:
https://gist.github.com/antonfirsov/01bb6b7b8569027cd71988ff33c66228#file-updated-output-32bit-3-1-txt

While output for dotnet run -c Release -f net6.0 --runtime win-x86 is:
https://gist.github.com/antonfirsov/01bb6b7b8569027cd71988ff33c66228#file-updated-output-32bit-6-0-txt

This looks like a regression to me.

[jkotas]

This was introduced by https://github.com/dotnet/runtime/pull/36152/files#diff-0edd676d0ef86c3df8cd23f712913c82d9106d1ea944d62e34189534bb817a5fL402-L410

Before this change, we treated the virtual < physical case as restricted environment and used the virtual memory limit as the limit. IIRC, I made this change since the restricted environment turns on hard_limit that makes the system behave quite differently, not how one would expect in an environment that is not actually the restricted container-like environment.

@Maoni0 Do you think we should go back to treating the virtual < physical case as restricted environment, or should we just cap the physical memory limit by the available virtual space?

[Maoni0]

oh interesting, I remember reviewing your PR @jkotas but I don't actually remember this particular part. maybe I missed it (if we did have a discussion about it I certainly don't remember now..). I think we should keep the previous behavior because when we get the memory load we'd want to get it based on the restricted_limit otherwise on a machine with a lot of physical mem we could be getting very low memory load but for the 32-bit process we want to get it based on the virtual.

[jkotas]

I agree that we should use the virtual address space as the limit.

The part that I have doubts about is whether is_restricted_physical_mem should be set to true when virtual_address_space < physical_memory. Do you think we should be setting is_restricted_physical_mem to true in this case? (It was the original behavior that I found questionable.)

[Maoni0]

looking at the code before that PR, GCToOSInterface::GetPhysicalMemoryLimit would set is_restricted to false for 32-bit processes with more physical memory than virtual mem (g_UseRestrictedVirtualMemory would be true) -

        if (is_restricted
#ifndef TARGET_UNIX
            && !g_UseRestrictedVirtualMemory
#endif
            )
            *is_restricted = true;

is_restricted didn't matter for 32-bit processes anyway since hard_limit is only turned on for 64-bit processes.