runtime: reported memory usage discrepancy (61%) between Go runtime and OS

[karalabe]

What version of Go are you using (go version)?

$ go version
1.12.1

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

Alpine docker container on top of Ubuntu.

What did you do?

After updating to Go 1.12, our memory usage pattern changed quite a lot. The amount of memory held by Go for the heap seemingly went down a bit and we can see it being relinquished more aggressively, if we are plotting runtime.MemStats.HeapSys - runtime.MemStats.HeapReleased .

The light purple and light green lines are the heap usage as reported by runtime.MemStats.HeapUsed (light purple = Go 1.12.1, light green = Go 1.11.5). The purple one uses 512MB more memory (not exactly the same code underneath), but that's not so relevant here.

The dark purple (Go 1.12.1) and dark blue (Go 1.11.5) lines are the heap total as reported by HeapSys - HeapReleased. This cleanly shows that Go 1.12 is more aggressive, as the "wasted heap" is a lot less compared to the used heap + relinquishing is also jaggedy compared to Go 1.11.5's flat curve.

If however we plot the amount of memory held by Go as seen by the OS (e.g. top, free or other process utilities), we get a 61% increase compared to Go 1.11.5. The top chart below (Memory (Master)) is Go 1.11.5's memory usage as reported by the OS for the same interval as the above heap charts. The bottom chart (Memory (EXP)) is Go 1.12.1's memory usage.

You can clearly see that Go 1.11.5's heap report matches with the OS's memory usage report. However, Go 1.12.1 reports vastly less memory being used than the OS reports. Based on our own production code, I tend to believe that the Go 1.12.1 heap charts are correct (i.e. they are in line with the 1.11.5 ones), however, this means Go is keeping hold of something that it itself is not aware of.

[ALTree]

1.12 release notes:

On Linux, the runtime now uses MADV_FREE to release unused memory. This is more efficient but may result in higher reported RSS. The kernel will reclaim the unused data when it is needed. To revert to the Go 1.11 behavior (MADV_DONTNEED), set the environment variable GODEBUG=madvdontneed=1.

Does the old behaviour return if you set GODEBUG=madvdontneed=1?

[karalabe]

Ah, didn't know about this. That might just explain it. Is there a way to ask the OS how much memory is in such a limbo state?

[ALTree]

Is there a way to ask the OS how much memory is in such a limbo state?

I personally don't know one, but this is more a Linux question than a Go one.

The runtime change is well documented in the release notes and the GODEBUG flag lets you override the new behaviour; so I don't think there's anything more to do here. I suggest we close this issue, unless you think this should be better documented and want to turn this into a doc bug.

[agnivade]

Seems like this is resolved. Please feel free to reopen if you think we need to document this somewhere else too.