runtime: use MADV_FREE on linux as well #23687
Comments
|
/cc @aclements |
bradfitz
added
the
NeedsInvestigation
ianlancetaylor
added
help wanted
NeedsFix
gopherbot
removed
the
NeedsInvestigation
ianlancetaylor
added
Performance
NeedsInvestigation
gopherbot
removed
the
NeedsFix
|
Change https://golang.org/cl/135395 mentions this issue: |
|
Hi all. I wondering if there is any data backing up claims about MADV_FREE actually improving anything. Asking because: a) I have not seen any data pointing that it does help (I co-maintain gperftools and over there we're defaulting MADV_FREE to off on Linux). b) there are numerous questions about implementation of this feature in kernel. First, while it is somewhat cheaper then MADV_DONTNEED, it is not that cheap, since page tables have to be walked and modified. It doesn't get reflected in RSS and thus interaction of this and memory reclaim facility in kernel, especially in settings with heavy usage of control groups isn't trivial. And also defaults about how this memory is preferred compared other memory during reclaim is also not entirely clear. Since this feature was lobbied by jemalloc folks, maybe there are some Facebook folks involved in golang who do have some data? I think this ticket should be re-opened. We should either gather and publish the data and see that indeed it helps, or revert defaults to off. IMHO. |
|
@alk I'm curious if you have tried any experiments with
Separately, given the volume of issues, I think some people automatically filter out notifications from closed issues, so you might want to open a new issue with your comments above. |
|
Change https://golang.org/cl/267100 mentions this issue: |
In Go 1.12, we changed the runtime to use MADV_FREE when available on Linux (falling back to MADV_DONTNEED) in CL 135395 to address issue #23687. While MADV_FREE is somewhat faster than MADV_DONTNEED, it doesn't affect many of the statistics that MADV_DONTNEED does until the memory is actually reclaimed under OS memory pressure. This generally leads to poor user experience, like confusing stats in top and other monitoring tools; and bad integration with management systems that respond to memory usage. We've seen numerous issues about this user experience, including #41818, #39295, #37585, #33376, and #30904, many questions on Go mailing lists, and requests for mechanisms to change this behavior at run-time, such as #40870. There are also issues that may be a result of this, but root-causing it can be difficult, such as #41444 and #39174. And there's some evidence it may even be incompatible with Android's process management in #37569. This CL changes the default to prefer MADV_DONTNEED over MADV_FREE, to favor user-friendliness and minimal surprise over performance. I think it's become clear that Linux's implementation of MADV_FREE ultimately doesn't meet our needs. We've also made many improvements to the scavenger since Go 1.12. In particular, it is now far more prompt and it is self-paced, so it will simply trickle memory back to the system a little more slowly with this change. This can still be overridden by setting GODEBUG=madvdontneed=0. Fixes #42330 (meta-issue). Fixes #41818, #39295, #37585, #33376, #30904 (many of which were already closed as "working as intended"). Change-Id: Ib6aa7f2dc8419b32516cc5a5fc402faf576c92e4 Reviewed-on: https://go-review.googlesource.com/c/go/+/267100 Trust: Austin Clements <[email protected]> Reviewed-by: Michael Knyszek <[email protected]>
On linux currently sysUnused uses
madvise(MADV_DONTNEED)to signal the kernel that a range of allocated memory contains unneeded data. After a successful call, the range (but not the data it contained before the call tomadvise) is still available but the first access to that range will unconditionally incur a page fault (needed to 0-fill the range).A potentially faster alternative is
MADV_FREE, available since linux 4.5 (and already used for bsd). The mechanism is very similar, but the page fault will only be incurred if the kernel, between the call tomadviseand the first access, decides to reuse that memory for something else.Off the top of my head, there are two ways to support both pre-4.5 and 4.5+ kernels:
MADV_FREEfirst andMADV_DONTNEEDifMADV_FREEfails): this is e.g. what node doesmadvise(MADV_FREE)on an allocated but unused range)The text was updated successfully, but these errors were encountered: