Memory leak in using slice of channels

[oghaffarinia]

What version of Go are you using (go version)?

$ go version
go version go1.15.6 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/omid/.cache/go-build"
GOENV="/home/omid/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/omid/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/omid/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.15"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.15/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build972503043=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Creating many channels inside slice (and not using them), garbage collector does not free them
This is the code I run: https://play.golang.org/p/0OaEXpZukQ9

What did you expect to see?

I expect to see memory usage be dropped when not using created objects

What did you see instead?

I see that memory usage never goes down and more memory is reserved for each iteration (i.e. there is memory leak)

[ALTree]

Please explain how exactly are you determining the amount of memory effectively used by the process, and thus that there's a leak.

Keep in mind that Go 1.15 on linux uses MADV_FREE to return memory, so the usual top/htop statistics will not be updated unless the OS actually reclaims the memory. But the Go program is not actually reserving the amount of memory shown. See #42330.

[oghaffarinia]

This is how I determine amount of memory usage: (This code is run)

$ go run test-leak.go
Enter length (quit to exit): 

$ ps --pid=14499 -o size
 SIZE
102652

Enter length (quit to exit): 100000
Enter length (quit to exit): 

$ ps --pid=14499 -o size
 SIZE
10827872

Enter length (quit to exit): 100000
Enter length (quit to exit): 100000
Enter length (quit to exit): 

$ ps --pid=14499 -o size
 SIZE
17403580

And continuing this way, the memory usage keeps going up until the whole RAM is used and process is killed by kernel

$ go run test-leak.go
Enter length (quit to exit): 100000
Enter length (quit to exit): 100000
Enter length (quit to exit): 100000
signal: killed

I have 16GB RAM + 8GB swap which was fully consumed by entering 100000 for 3 times

[seankhliao]

If you add explicit runtime.GC() calls it should work closer as to what you expect, only keeping ~10G of memory. leave it long enough and it clears out everything, which indicates it's not a leak, just GC not happening yet.

[hawell]

you're allocating too much memory too fast. the scavenger wont return unused memory back to os until after a certain amount of time. (this used to be 5 minutes)
if you wait long enough (2 minutes for forced GC at most) you will that the memory is freed.
you can check this by using GODEBUG=gctrace=1
one more thing: this has nothing to do with channels
as @seankhliao mentioned you can use runtime.GC() (or debug.FreeOSMemory() ) explicitly

[randall77]

Channels are 96 bytes, plus 8 for the entry in the all array. You're allocating 100M of them, which is ~10GB of live data. Go will allocate memory 2x that (with GOGC=100) for its heap, so you will need ~20GB to run. This is the memory usage I see when running in a loop forever. I suspect it fails on your machine on the 3rd iteration just because of how Go grows the heap towards that goal size.

Get a larger machine, or set GOGC to a smaller value. Running runtime.GC explicitly is kinda like setting GOGC to a smaller value, if you can manage to call it at the right time (when the all slice is not live).

I'm going to close, as I don't think there's a bug here. If you can demonstrate otherwise (heap growing to >2x live data), please reopen.

[oghaffarinia]

Thanks for explaining, calling runtime.GC() solves the problem, and for some reason the automatic garbage collection does not take place for a long running program (even running for several days).
I will update this if I can demonstrate the problem more precisely, thanks anyway