net/http: server will drop tcp connections after a certain number of concurrent connections are made

[matthewdavidrodgers]

What version of Go are you using (go version)?

Tested on two machines:

arm:

$ go version
go version go1.20.6 darwin/arm64

x86:

$ go version
go version go1.20.6 linux/amd64

Does this issue reproduce with the latest release?

yes. tested on go version devel go1.22-8ecdc4e Thu Aug 10 14:38:22 2023 +0000 darwin/arm64 as well

What operating system and processor architecture are you using (go env)?

go env Output

arm:

GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/matthewrodgers/Library/Caches/go-build"
GOENV="/Users/matthewrodgers/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/matthewrodgers/go/pkg/mod"
GONOPROXY="code.cfops.it"
GONOSUMDB="code.cfops.it"
GOOS="darwin"
GOPATH="/Users/matthewrodgers/go"
GOPRIVATE="code.cfops.it"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.20.6/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.20.6/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.6"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/d9/b6xts_292jl1xm48py971tm40000gp/T/go-build4134932022=/tmp/go-build -gno-record-gcc-switches -fno-common"

x86:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/matthewrodgers/.cache/go-build"
GOENV="/home/matthewrodgers/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/matthewrodgers/go/pkg/mod"
GONOPROXY="code.cfops.it"
GONOSUMDB="code.cfops.it"
GOOS="linux"
GOPATH="/home/matthewrodgers/go"
GOPRIVATE="code.cfops.it"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.6"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3084080192=/tmp/go-build -gno-record-gcc-switches"

What did you do?

ran this test:

package main

import (
	"fmt"
	"io"
	"net"
	"net/http"
	"testing"
)

type testserv struct {
	t *testing.T
}

func (ss *testserv) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK)
	_, err := io.WriteString(w, "OK")
	if err != nil {
		ss.t.Fatal("could not start server: ", err)
	}
}

func TestConcurrentConnections(t *testing.T) {
	for _, conns := range []int{100, 1000} {
		t.Run(fmt.Sprintf("%d connections", conns), func(t *testing.T) {
			completed := make(chan struct{}, conns)
			failed := make(chan error)

			listener, err := net.Listen("tcp", "127.0.0.1:0")
			if err != nil {
				t.Fatal("could not list to localhost: ", err)
			}
			defer listener.Close()

			go http.Serve(listener, &testserv{t: t})

			// make concurrent requests to the server
			for i := 0; i < conns; i++ {
				go func() {
					_, err := http.Get("http://" + listener.Addr().String())
					// if the request failed, report back to main goroutine
					if err != nil {
						failed <- err
					}
					completed <- struct{}{}
				}()
			}

			completedCount := 0
			for {
				select {
				case err := <-failed:
					t.Fatal(fmt.Sprintf("completed %d connections but got err: %v", completedCount, err))
				case <-completed:
					completedCount += 1
				}

				if completedCount == conns {
					break
				}
			}
		})
	}
}

What did you expect to see?

both test cases, opening 100 concurrent connections to a server, and opening 1000 concurrent connections to a server, succeed

What did you see instead?

on x64, both test cases pass as expected
on arm, only the 100 concurrent connections case succeeds - the 1000 concurrent connections case fails with:
completed 0 connections but got err: Get "http://127.0.0.1:56934": read tcp 127.0.0.1:56943->127.0.0.1:56934: read: connection reset by peer, with completed connections varying slightly. from my testing, something around 130 concurrent connections triggers this behavior

[thanm]

I think this may be a darwin problem as opposed to a linux problem; when I run your test on linux/arm64 I don't see any issues. Perhaps macos is imposing some sort of resource limit in this case?

[mknyszek]

CC @neild

[panjf2000]

I think this has something to do with #54529

Try this solution, it ought to work.

[thanm]

Thanks @panjf2000 that is helpful. I am going to close this as a duplicate of #54529.

[thanm]

Duplicate of #54529.