runtime

[harsh8118]

What version of Go are you using (go version)?

$ go version
1.19.3

Does this issue reproduce with the latest release?

randomly reproducible

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
harshg@ss1:~$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/harshg/.cache/go-build"
GOENV="/home/harshg/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/harshg/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/harshg/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1325167941=/tmp/go-build -gno-record-gcc-switches"
harshg@ss1:~$

What did you do?

program crashed silently in idle condition after 10 hours

What did you expect to see?

no crash

What did you see instead?

2023-05-15T00:29:00.060915154Z stderr F goroutine 284 [select (scan), 572 minutes]:
2023-05-15T00:29:00.060935029Z stderr F fatal error: unexpected signal during runtime execution
2023-05-15T00:29:00.060945423Z stderr F panic during panic
2023-05-15T00:29:00.060953584Z stderr F [signal SIGSEGV: segmentation violation code=0x1 addr=0x118 pc=0x45f1fc]
2023-05-15T00:29:00.060960853Z stderr F
2023-05-15T00:29:00.060968254Z stderr F runtime stack:
2023-05-15T00:29:00.06097619Z stderr F runtime.throw({0xf084e8?, 0x1679de0?})
2023-05-15T00:29:00.060983898Z stderr F /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f6f019e2bc8 sp=0x7f6f019e2b98 pc=0x438a5d
2023-05-15T00:29:00.060991749Z stderr F runtime.sigpanic()
2023-05-15T00:29:00.060999421Z stderr F /usr/local/go/src/runtime/signal_unix.go:821 +0x3e9 fp=0x7f6f019e2c28 sp=0x7f6f019e2bc8 pc=0x44f8c9
2023-05-15T00:29:00.061021651Z stderr F runtime.gentraceback(0x145a250?, 0x1679de0?, 0x7f6f019e2fe0?, 0xc0002ab520, 0x0, 0x0, 0x64, 0x0, 0x7f6f019e3000?, 0x0)
2023-05-15T00:29:00.061031352Z stderr F /usr/local/go/src/runtime/traceback.go:258 +0x8bc fp=0x7f6f019e2f80 sp=0x7f6f019e2c28 pc=0x45f1fc
2023-05-15T00:29:00.061039Z stderr F runtime.traceback1(0xc0002ab520?, 0x43a500?, 0x3?, 0xc0002ab520, 0x461ffb?)
2023-05-15T00:29:00.061047254Z stderr F /usr/local/go/src/runtime/traceback.go:776 +0x1b6 fp=0x7f6f019e3140 sp=0x7f6f019e2f80 pc=0x4612f6
2023-05-15T00:29:00.061055129Z stderr F runtime.traceback(...)
2023-05-15T00:29:00.061062962Z stderr F /usr/local/go/src/runtime/traceback.go:723
2023-05-15T00:29:00.061070644Z stderr F runtime.tracebackothers.func1(0xc0002ab520)
2023-05-15T00:29:00.061088677Z stderr F /usr/local/go/src/runtime/traceback.go:992 +0xe5 fp=0x7f6f019e3180 sp=0x7f6f019e3140 pc=0x462225
2023-05-15T00:29:00.061098674Z stderr F runtime.forEachGRace(0x7f6f019e31e8)
2023-05-15T00:29:00.061116638Z stderr F /usr/local/go/src/runtime/proc.go:604 +0x4d fp=0x7f6f019e31b0 sp=0x7f6f019e3180 pc=0x43c2ad
2023-05-15T00:29:00.061126519Z stderr F runtime.tracebackothers(0xc000500680?)
2023-05-15T00:29:00.061134431Z stderr F /usr/local/go/src/runtime/traceback.go:978 +0xe5 fp=0x7f6f019e3218 sp=0x7f6f019e31b0 pc=0x462105
2023-05-15T00:29:00.061150825Z stderr F runtime.dopanic_m(0xc000500680, 0x1022668?, 0x1?)
2023-05-15T00:29:00.061159184Z stderr F /usr/local/go/src/runtime/panic.go:1269 +0x285 fp=0x7f6f019e3290 sp=0x7f6f019e3218 pc=0x439405
2023-05-15T00:29:00.061167318Z stderr F runtime.fatalthrow.func1()
2023-05-15T00:29:00.061189246Z stderr F /usr/local/go/src/runtime/panic.go:1123 +0x48 fp=0x7f6f019e32d0 sp=0x7f6f019e3290 pc=0x438e28
2023-05-15T00:29:00.061206685Z stderr F runtime.fatalthrow(0x19e3318?)
2023-05-15T00:29:00.061230581Z stderr F /usr/local/go/src/runtime/panic.go:1120 +0x6c fp=0x7f6f019e3310 sp=0x7f6f019e32d0 pc=0x438dac
2023-05-15T00:29:00.061245611Z stderr F runtime.throw({0xee5636?, 0x1679de0?})
2023-05-15T00:29:00.061253829Z stderr F /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f6f019e3340 sp=0x7f6f019e3310 pc=0x438a5d
2023-05-15T00:29:00.061276212Z stderr F runtime.gentraceback(0x10?, 0x10225a0?, 0x0?, 0xc0002ab520, 0x0, 0x0, 0x7fffffff, 0x7f6f019e3878, 0x0?, 0x0)
2023-05-15T00:29:00.06128627Z stderr F /usr/local/go/src/runtime/traceback.go:270 +0x1bb0 fp=0x7f6f019e3698 sp=0x7f6f019e3340 pc=0x4604f0
2023-05-15T00:29:00.061294495Z stderr F runtime.scanstack(0xc0002ab520, 0xc000053238)
2023-05-15T00:29:00.061302225Z stderr F /usr/local/go/src/runtime/mgcmark.go:804 +0x1f2 fp=0x7f6f019e38a0 sp=0x7f6f019e3698 pc=0x4203b2
2023-05-15T00:29:00.061309992Z stderr F runtime.markroot.func1()
2023-05-15T00:29:00.061325289Z stderr F /usr/local/go/src/runtime/mgcmark.go:239 +0xb5 fp=0x7f6f019e38f0 sp=0x7f6f019e38a0 pc=0x41f1b5
2023-05-15T00:29:00.061334738Z stderr F runtime.markroot(0xc000053238, 0x67, 0x1)
2023-05-15T00:29:00.061351814Z stderr F /usr/local/go/src/runtime/mgcmark.go:213 +0x1a5 fp=0x7f6f019e3990 sp=0x7f6f019e38f0 pc=0x41ee65
2023-05-15T00:29:00.061361313Z stderr F runtime.gcDrain(0xc000053238, 0x3)
2023-05-15T00:29:00.061368833Z stderr F /usr/local/go/src/runtime/mgcmark.go:1069 +0x39f fp=0x7f6f019e39f0 sp=0x7f6f019e3990 pc=0x420f5f
2023-05-15T00:29:00.061381907Z stderr F runtime.gcBgMarkWorker.func2()
2023-05-15T00:29:00.061389941Z stderr F /usr/local/go/src/runtime/mgc.go:1348 +0xad fp=0x7f6f019e3a40 sp=0x7f6f019e39f0 pc=0x41d46d
2023-05-15T00:29:00.061397373Z stderr F runtime.systemstack()
2023-05-15T00:29:00.061405054Z stderr F /usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x7f6f019e3a48 sp=0x7f6f019e3a40 pc=0x46c509
2023-05-15T00:29:00.069193134Z stderr F [signal SIGSEGV: segmentation violation code=0x2 addr=0xc002000000 pc=0x46f638]
2023-05-15T00:29:00.069230008Z stderr F
2023-05-15T00:29:00.069238773Z stderr F runtime stack:
2023-05-15T00:29:00.069247004Z stderr F runtime.throw({0xf084e8?, 0x1755820?})
2023-05-15T00:29:00.069255274Z stderr F /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f6f016a4418 sp=0x7f6f016a43e8 pc=0x438a5d
2023-05-15T00:29:00.069263488Z stderr F runtime.sigpanic()
2023-05-15T00:29:00.069271819Z stderr F /usr/local/go/src/runtime/signal_unix.go:821 +0x3e9 fp=0x7f6f016a4478 sp=0x7f6f016a4418 pc=0x44f8c9
2023-05-15T00:29:00.069279502Z stderr F runtime.memmove()
2023-05-15T00:29:00.069287077Z stderr F /usr/local/go/src/runtime/memmove_amd64.s:111 +0xb8 fp=0x7f6f016a4480 sp=0x7f6f016a4478 pc=0x46f638
2023-05-15T00:29:00.069295053Z stderr F runtime.syncadjustsudogs(0xc0002aa4e0, 0x708, 0x7f6f016a4548)
2023-05-15T00:29:00.069302566Z stderr F /usr/local/go/src/runtime/stack.go:842 +0xeb fp=0x7f6f016a44c0 sp=0x7f6f016a4480 pc=0x4524ab
2023-05-15T00:29:00.069310029Z stderr F runtime.copystack(0xc0002aa4e0, 0x7f6f016a4840?)
2023-05-15T00:29:00.069317444Z stderr F /usr/local/go/src/runtime/stack.go:911 +0x375 fp=0x7f6f016a4678 sp=0x7f6f016a44c0 pc=0x4528b5
2023-05-15T00:29:00.069324854Z stderr F runtime.shrinkstack(0xc0002aa4e0)
2023-05-15T00:29:00.06933255Z stderr F /usr/local/go/src/runtime/stack.go:1216 +0x134 fp=0x7f6f016a4698 sp=0x7f6f016a4678 pc=0x4536f4
2023-05-15T00:29:00.069340138Z stderr F runtime.scanstack(0xc0002aa4e0, 0xc000055738)
2023-05-15T00:29:00.069347688Z stderr F /usr/local/go/src/runtime/mgcmark.go:775 +0xf2 fp=0x7f6f016a48a0 sp=0x7f6f016a4698 pc=0x4202b2
2023-05-15T00:29:00.069378986Z stderr F runtime.markroot.func1()
2023-05-15T00:29:00.069387273Z stderr F /usr/local/go/src/runtime/mgcmark.go:239 +0xb5 fp=0x7f6f016a48f0 sp=0x7f6f016a48a0 pc=0x41f1b5
2023-05-15T00:29:00.069394738Z stderr F runtime.markroot(0xc000055738, 0x61, 0x1)
2023-05-15T00:29:00.069402127Z stderr F /usr/local/go/src/runtime/mgcmark.go:213 +0x1a5 fp=0x7f6f016a4990 sp=0x7f6f016a48f0 pc=0x41ee65
2023-05-15T00:29:00.069409569Z stderr F runtime.gcDrain(0xc000055738, 0x3)
2023-05-15T00:29:00.069417074Z stderr F /usr/local/go/src/runtime/mgcmark.go:1069 +0x39f fp=0x7f6f016a49f0 sp=0x7f6f016a4990 pc=0x420f5f
2023-05-15T00:29:00.069425208Z stderr F runtime.gcBgMarkWorker.func2()
2023-05-15T00:29:00.069432913Z stderr F /usr/local/go/src/runtime/mgc.go:1348 +0xad fp=0x7f6f016a4a40 sp=0x7f6f016a49f0 pc=0x41d46d
2023-05-15T00:29:00.069440483Z stderr F runtime.systemstack()
2023-05-15T00:29:00.069447934Z stderr F /usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x7f6f016a4a48 sp=0x7f6f016a4a40 pc=0x46c509

2023-05-15T00:29:00.036892142Z stderr F runtime: g 284: unexpected return pc for runtime.gopark called from 0x200a000020f18212
2023-05-15T00:29:00.03694504Z stderr F stack: frame={sp:0xc0013eadb8, fp:0xc0013eadd8} stack=[0xc0013ea000,0xc0013eb000)
2023-05-15T00:29:00.03695504Z stderr F 0x000000c0013eacb8: 0x038205b605b501d7 0x05af06d905b73837
2023-05-15T00:29:00.036963002Z stderr F 0x000000c0013eacc8: 0x06d506dd05b105b0 0x0208061a06c306c2
2023-05-15T00:29:00.036970872Z stderr F 0x000000c0013eacd8: 0x07123a1203c08018 0x0a000020809855ab
2023-05-15T00:29:00.036979047Z stderr F 0x000000c0013eace8: 0x8903880387038629 0x5f305e5d5c5f3003
2023-05-15T00:29:00.036986783Z stderr F 0x000000c0013eacf8: 0x333231305e5d6160 0x038205b605b53534
2023-05-15T00:29:00.036994652Z stderr F 0x000000c0013ead08: 0x05af06d905b73837 0x30180208041a05b0
2023-05-15T00:29:00.037002375Z stderr F 0x000000c0013ead18: 0xb001aad809121912 0xa0050a000001c083
2023-05-15T00:29:00.037028576Z stderr F 0x000000c0013ead28: 0x0208051a1e029604 0xac09121e12019018
2023-05-15T00:29:00.037039009Z stderr F 0x000000c0013ead38: 0x0000018080e002d5 0x5958575601cd0b0a
2023-05-15T00:29:00.037047001Z stderr F 0x000000c0013ead48: 0x08041a0b0a090807 0x06e1081222301802
2023-05-15T00:29:00.037054956Z stderr F 0x000000c0013ead58: 0x08062205d9a1e418 0x12011001f610029d
2023-05-15T00:29:00.037062637Z stderr F 0x000000c0013ead68: 0x80c010808409121b 0x06e1080a00000280
2023-05-15T00:29:00.037070568Z stderr F 0x000000c0013ead78: 0x041a02bd02bc03d6 0xe2081a2210180208
2023-05-15T00:29:00.03707797Z stderr F 0x000000c0013ead88: 0x0622068e88ba1806 0x062202d810038608
2023-05-15T00:29:00.037085574Z stderr F 0x000000c0013ead98: 0x011016861003da08 0x21e48a0906122a12
2023-05-15T00:29:00.037093072Z stderr F 0x000000c0013eada8: 0x06c506c4180a0000 0x06d906c806c706c6
2023-05-15T00:29:00.03711106Z stderr F 0x000000c0013eadb8: <0x06e205b105b005af 0x0208061a06c306c2
2023-05-15T00:29:00.037121224Z stderr F 0x000000c0013eadc8: 0x0612321203c08018 !0x200a000020f18212
2023-05-15T00:29:00.037128938Z stderr F 0x000000c0013eadd8: >0x01d701d601d501d4 0x3837038205b605b5
2023-05-15T00:29:00.03713688Z stderr F 0x000000c0013eade8: 0x05b005af06d905b7 0x06c306c206e205b1
2023-05-15T00:29:00.037144731Z stderr F 0x000000c0013eadf8: 0x01e080180208061a 0x80a8339a07123912
2023-05-15T00:29:00.037152468Z stderr F 0x000000c0013eae08: 0x5f03d4280a000020 0x5d61605f305e5d5c
2023-05-15T00:29:00.03715992Z stderr F 0x000000c0013eae18: 0xb53534333231305e 0xb73837038205b605
2023-05-15T00:29:00.037167459Z stderr F 0x000000c0013eae28: 0xb105b005af06d905 0x08041a06c206e205
2023-05-15T00:29:00.037174966Z stderr F 0x000000c0013eae38: 0x9a07123a12501802 0x290a00002080a833
2023-05-15T00:29:00.037182535Z stderr F 0x000000c0013eae48: 0x02df171604920488 0xbc05bb05ba191816
2023-05-15T00:29:00.037190001Z stderr F 0x000000c0013eae58: 0xc005bf05be05bd05 0xc405c305c205c105
2023-05-15T00:29:00.03719735Z stderr F 0x000000c0013eae68: 0xc306c206e205b105 0x1250180208041a06
2023-05-15T00:29:00.037204736Z stderr F 0x000000c0013eae78: 0x2085c002f4071223 0xb93901ce110a0000
2023-05-15T00:29:00.037222153Z stderr F 0x000000c0013eae88: 0xe205b105b005af05 0x08051a06c306c206
2023-05-15T00:29:00.037231828Z stderr F 0x000000c0013eae98: 0x071229120b801802 0x0a00002085c002f4
2023-05-15T00:29:00.037239439Z stderr F 0x000000c0013eaea8: 0x373635343304bc17 0x05b005af05b93938
2023-05-15T00:29:00.03727113Z stderr F 0x000000c0013eaeb8: 0x06c306c206e205b1 0x120b80180208051a
2023-05-15T00:29:00.037279858Z stderr F 0x000000c0013eaec8: 0x8088028080081239 0x980397270a000020
2023-05-15T00:29:00.037287863Z stderr F fatal error: unknown caller pc
2023-05-15T00:29:00.045003905Z stderr F fatal error: unexpected signal during runtime execution
2023-05-15T00:29:00.04645483Z stderr F
2023-05-15T00:29:00.046472903Z stderr F runtime stack:
2023-05-15T00:29:00.046490843Z stderr F runtime.throw({0xee5636?, 0x1679de0?})
2023-05-15T00:29:00.04650042Z stderr F /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f6f019e3340 sp=0x7f6f019e3310 pc=0x438a5d
2023-05-15T00:29:00.046558481Z stderr F runtime.gentraceback(0x10?, 0x10225a0?, 0x0?, 0xc0002ab520, 0x0, 0x0, 0x7fffffff, 0x7f6f019e3878, 0x0?, 0x0)
2023-05-15T00:29:00.046582615Z stderr F /usr/local/go/src/runtime/traceback.go:270 +0x1bb0 fp=0x7f6f019e3698 sp=0x7f6f019e3340 pc=0x4604f0
2023-05-15T00:29:00.04659951Z stderr F runtime.scanstack(0xc0002ab520, 0xc000053238)
2023-05-15T00:29:00.046621212Z stderr F /usr/local/go/src/runtime/mgcmark.go:804 +0x1f2 fp=0x7f6f019e38a0 sp=0x7f6f019e3698 pc=0x4203b2
2023-05-15T00:29:00.04664642Z stderr F runtime.markroot.func1()
2023-05-15T00:29:00.046657147Z stderr F /usr/local/go/src/runtime/mgcmark.go:239 +0xb5 fp=0x7f6f019e38f0 sp=0x7f6f019e38a0 pc=0x41f1b5
2023-05-15T00:29:00.046665083Z stderr F runtime.markroot(0xc000053238, 0x67, 0x1)
2023-05-15T00:29:00.046672924Z stderr F /usr/local/go/src/runtime/mgcmark.go:213 +0x1a5 fp=0x7f6f019e3990 sp=0x7f6f019e38f0 pc=0x41ee65
2023-05-15T00:29:00.046706003Z stderr F runtime.gcDrain(0xc000053238, 0x3)
2023-05-15T00:29:00.046738992Z stderr F /usr/local/go/src/runtime/mgcmark.go:1069 +0x39f fp=0x7f6f019e39f0 sp=0x7f6f019e3990 pc=0x420f5f
2023-05-15T00:29:00.046766357Z stderr F runtime.gcBgMarkWorker.func2()
2023-05-15T00:29:00.046777442Z stderr F /usr/local/go/src/runtime/mgc.go:1348 +0xad fp=0x7f6f019e3a40 sp=0x7f6f019e39f0 pc=0x41d46d
2023-05-15T00:29:00.046786034Z stderr F runtime.systemstack()
2023-05-15T00:29:00.046794051Z stderr F /usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x7f6f019e3a48 sp=0x7f6f019e3a40 pc=0x46c509
2023-05-15T00:29:00.046800968Z stderr F

[bcmills]

Looks like heap corruption. What steps have you taken to rule out a bug in the program, such as a stray write (via cgo or unsafe) or a data race?

[harsh8118]

hi i am new to debuggin core in go world. can you suggest me what next steps should i take to debug it further.
we are not using cgo plugin. also verified using data race detector there is no race in my program.

[ianlancetaylor]

Is the problem repeatable?

Is there a way that we can reproduce it ourselves?

It does look like heap corruption. If it's repeatable, look closely at any uses of the unsafe package in your program.

[harsh8118]

this issue is not easily reproducible. we are hitting this issue in deployment when it runs for 20+ hrs that also not every time.
i am not using any unsafe package in the program.
it is a complex networking application and this is running 130+ goroutines parallelly.

[harsh8118]

@bcmills @evanphx any update on this. please suggest how it is help corruption and not stack corruption.

[harsh8118]

adding traces from dlv

(dlv) bt
0 0x0000000000470041 in runtime.raise
at /usr/local/go/src/runtime/sys_linux_amd64.s:154
1 0x000000000046c4a0 in runtime.systemstack_switch
at /usr/local/go/src/runtime/asm_amd64.s:463
2 0x000000000041d105 in runtime.gcBgMarkWorker
at /usr/local/go/src/runtime/mgc.go:1335
3 0x000000000046e6c1 in runtime.goexit
at /usr/local/go/src/runtime/asm_amd64.s:1598

ntrolBuffer).get (0x830d35) [select 29286134242491]
Goroutine 2319 - User: /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:1144 google.golang.org/grpc/internal/transport.(
*http2Server).keepalive (0x8505f3) [select 29286134242491]
Goroutine 2320 - User: /usr/local/go/src/net/fd_posix.go:55 net.(*netFD).Read (0x548369) [IO wait 29286134242491]
Goroutine 3417510164015836018 - User: :0 ??? (0x676e452f6d6f632e)
Goroutine 7094691637185961010 - User: :0 ??? (0x7465536966615369)
Goroutine 7308895138221469549 - User: :0 ??? (0x656e656d65727478)
Goroutine 7441690888330176367 - User: :0 ??? (0x6f632f736f2d6172)
Goroutine 7441690888330176367 - User: :0 ??? (0x6f632f736f2d6172)
Goroutine 7598247007645951854 - User: :0 ??? (0x6d6f632e736b726f)
Goroutine 8247349193587455336 - User: :0 ??? (0x6e65646e496c6168)
Goroutine 8382155981719168342 - User: :0 ??? (0x7365636f72702e62)
Goroutine 8389723868813943908 - User: :0 ??? (0x692f736f2d617272)
Goroutine 8389723868813943908 - User: :0 ??? (0x692f736f2d617272)
[119 goroutines]

(dlv) gr 284
Switched from 282 to 284 (thread 28)
(dlv) stack
0 0x000000000043b776 in runtime.gopark
at /usr/local/go/src/runtime/proc.go:382
1 0x200a000020f18212 in ???
at ?:-1
error: error while reading spliced memory at 0x612321203c08020: EOF
(truncated)
(dlv)

[ianlancetaylor]

Have you run your program under the race detector?

[harsh8118]

@ianlancetaylor yes tried with race detector also, race detector didn't reported any issue.

[harsh8118]

in findObject() how to set debug.invalidptr to true ? will it help to debug the badpointer issue ?

[ianlancetaylor]

debug.invalidptr is 1 by default. You can disable it using GODEBUG=invalidptr=0 in the environment, but it's normally not a good idea. See https://pkg.go.dev/runtime.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)