Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: segmentation violation runtime.(*unwinder).next #66151

Closed
ben-swit opened this issue Mar 7, 2024 · 5 comments
Closed

runtime: segmentation violation runtime.(*unwinder).next #66151

ben-swit opened this issue Mar 7, 2024 · 5 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@ben-swit
Copy link

ben-swit commented Mar 7, 2024 •
edited
Loading

Go version

go 1.21 amd64/linux (golang:1.21.7-alpine3.19)

Output of go env in your module/workspace:

default env

What did you do?

occurs very intermittently in the operating environment.
I'm looking for a way to reproduce it.

What did you see happen?

SIGSEGV: segmentation violation
PC=0x464585 m=12 sigcode=1

goroutine 0 [idle]:
runtime.(*unwinder).next(0x7f858f9e9300)
/usr/local/go/src/runtime/traceback.go:460 +0x105 fp=0x7f858f9e9150 sp=0x7f858f9e90d8 pc=0x464585
runtime: g 0: unexpected return pc for runtime.(*unwinder).next called from 0x146f0
stack: frame={sp:0x7f858f9e90d8, fp:0x7f858f9e9150} stack=[0x7f858f9c9a40,0x7f858f9e9640)
0x00007f858f9e8fd8: 0x0000000000000000 0x00007f858f9e0000
0x00007f858f9e8fe8: 0x00007f858f9e9001 0x000000c00038b400
0x00007f858f9e8ff8: 0x0000000000000000 0x00007f858f9e0000
0x00007f858f9e9008: 0x0000000000000002 0x0000000000000000
0x00007f858f9e9018: 0x0000000000000000 0x0000000000000000
0x00007f858f9e9028: 0x0000000500000000 0x0000000003cee220
0x00007f858f9e9038: 0x000000000010faa3 0x0000000000000000
0x00007f858f9e9048: 0x00000000030eae47 0x0000000000862344 <golang.org/x/net/http2.(*Framer).ReadFrame+0x0000000000000084>
0x00007f858f9e9058: 0x0000000000000098 0x00000000004642f8 <runtime.(*unwinder).resolveInternal+0x0000000000000158>
0x00007f858f9e9068: 0xa44ec3bae88600f3 0x00007f858f9e90d8
0x00007f858f9e9078: 0x00007f858f9e9118 0x00000000004740c0 <runtime.asmcgocall+0x0000000000000000>
0x00007f858f9e9088: 0x0000000000000000 0x000000c000f541a0
0x00007f858f9e9098: 0x0000000000000001 0x00007f858f9e90f0
0x00007f858f9e90a8: 0x0000000000474166 <runtime.asmcgocall+0x00000000000000a6> 0x00000001035731e8
0x00007f858f9e90b8: 0x000000c0033ce000 0x0000000000000000
0x00007f858f9e90c8: 0x00007f858f9e9140 0x00000000004644cc <runtime.(*unwinder).next+0x000000000000004c>
0x00007f858f9e90d8: <0x00007f858f9e90f0 0x0000000000000000
0x00007f858f9e90e8: 0x000000c00910fe30 0x00007f858f9e9140
0x00007f858f9e90f8: 0x00000000000144a5 0x00007f85dd88e1c0
0x00007f858f9e9108: 0x00007f858f9e9118 0x0000000100000000
0x00007f858f9e9118: 0x00000000000149b0 0x00000000035731e8
0x00007f858f9e9128: 0x0000000003cee220 0x0000000000474000 <runtime.call1073741824+0x0000000000000060>
0x00007f858f9e9138: 0x000000c000603d40 0x00007f858f9e9478
0x00007f858f9e9148: 0x00000000000146f0 >0x00007f858f9e9300
0x00007f858f9e9158: 0x00007f858f9e91b0 0x000000000273f3f0
0x00007f858f9e9168: 0x0a1b0a676e702e58 0x000000c000603d40
0x00007f858f9e9178: 0x616e040a1c0a0100 0xeb84b2ebb49deca4
0x00007f858f9e9188: 0x0000000000000004 0x0000000000000a60
0x00007f858f9e9198: 0x0000000000000000 0x0000000000000000
0x00007f858f9e91a8: 0x0000000000000000 0x0000000000862344 <golang.org/x/net/http2.(*Framer).ReadFrame+0x0000000000000084>
0x00007f858f9e91b8: 0x000000000010fb7b 0x000000000076a0d7 <crypto/tls.(*Conn).Read+0x0000000000000157>
0x00007f858f9e91c8: 0x00000001000ee7fe 0x0000000000861c04 <golang.org/x/net/http2.readFrameHeader+0x0000000000000064>
0x00007f858f9e91d8: 0x000000010010f8d8 0x0000000000000000
0x00007f858f9e91e8: 0x0000000000000000 0x0000000000000000
0x00007f858f9e91f8: 0x0000000000000000 0x000000000057c5d6 <bufio.(*Reader).Read+0x0000000000000196>
0x00007f858f9e9208: 0x00000000000764f2 0x000000000046e5e4 <internal/poll.runtime_pollWait+0x0000000000000084>
0x00007f858f9e9218: 0x000000000002b655 0x00000000005180d7 <bytes.(*Buffer).ReadFrom+0x0000000000000097>
0x00007f858f9e9228: 0x000000000005ec32 0x00000000004e006f <io.ReadAtLeast+0x000000000000008f>
0x00007f858f9e9238: 0x000000000004e77d 0x00000000004e6d99 <internal/poll.(*FD).Read+0x0000000000000279>
0x00007f858f9e9248: 0x00000002000507d1

goroutine 19 [GC worker (active)]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc0000aef50 sp=0xc0000aef40 pc=0x4724c8
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1355 +0x1f2 fp=0xc0000aefe0 sp=0xc0000aef50 pc=0x421af2
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000aefe8 sp=0xc0000aefe0 pc=0x4744a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 1 [select, 850 minutes]:
runtime.gopark(0xc002b65e10?, 0x2?, 0x98?, 0x5a?, 0xc002b65c94?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00607da40 sp=0xc00607da20 pc=0x440dce
runtime.selectgo(0xc00607de10, 0xc002b65c90, 0x2?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00607db60 sp=0xc00607da40 pc=0x4512a5
main.main()
/go/src/swit/server/cmd/main.go:224 +0x1305 fp=0xc00607df40 sp=0xc00607db60 pc=0x1add985
runtime.main()
/usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc00607dfe0 sp=0xc00607df40 pc=0x44095b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00607dfe8 sp=0xc00607dfe0 pc=0x4744a1

goroutine 2 [force gc (idle), 66 minutes]:
runtime.gopark(0x36bec9c2e1c9f?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000b2fa8 sp=0xc0000b2f88 pc=0x440dce
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc0000b2fe0 sp=0xc0000b2fa8 pc=0x440c33
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000b2fe8 sp=0xc0000b2fe0 pc=0x4744a1
created by runtime.init.6 in goroutine 1
/usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait, 1 minutes]:
runtime.gopark(0x3ed7101?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000b3778 sp=0xc0000b3758 pc=0x440dce
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
/usr/local/go/src/runtime/mgcsweep.go:321 +0xdf fp=0xc0000b37c8 sp=0xc0000b3778 pc=0x42ad1f
runtime.gcenable.func1()
/usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000b37e0 sp=0xc0000b37c8 pc=0x41fe65
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000b37e8 sp=0xc0000b37e0 pc=0x4744a1
created by runtime.gcenable in goroutine 1
/usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait, 1 minutes]:
runtime.gopark(0x16527e?, 0x740dc?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000b3f70 sp=0xc0000b3f50 pc=0x440dce
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x3ef1a60)
/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc0000b3fa0 sp=0xc0000b3f70 pc=0x428589
runtime.bgscavenge(0x0?)
/usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc0000b3fc8 sp=0xc0000b3fa0 pc=0x428b39
runtime.gcenable.func2()
/usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc0000b3fe0 sp=0xc0000b3fc8 pc=0x41fe05
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000b3fe8 sp=0xc0000b3fe0 pc=0x4744a1
created by runtime.gcenable in goroutine 1
/usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 5 [finalizer wait, 1 minutes]:
runtime.gopark(0x0?, 0xc0058f5098?, 0x0?, 0x80?, 0x1000000010?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc009103e20 sp=0xc009103e00 pc=0x440dce
runtime.runfinq()
/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc009103fe0 sp=0xc009103e20 pc=0x41ee87
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc009103fe8 sp=0xc009103fe0 pc=0x4744a1
created by runtime.createfing in goroutine 1
/usr/local/go/src/runtime/mfinal.go:163 +0x3d

goroutine 6 [GC worker (idle), 2 minutes]:
runtime.gopark(0x3f2f780?, 0x1?, 0xec?, 0xa3?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000b4750 sp=0xc0000b4730 pc=0x440dce
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000b47e0 sp=0xc0000b4750 pc=0x4219e5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000b47e8 sp=0xc0000b47e0 pc=0x4744a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 7 [GC worker (idle), 1 minutes]:
runtime.gopark(0x36f61b2179539?, 0x3?, 0x46?, 0xea?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000b4f50 sp=0xc0000b4f30 pc=0x440dce
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000b4fe0 sp=0xc0000b4f50 pc=0x4219e5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000b4fe8 sp=0xc0000b4fe0 pc=0x4744a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x3f2f780?, 0x1?, 0x46?, 0x63?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000ae750 sp=0xc0000ae730 pc=0x440dce
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000ae7e0 sp=0xc0000ae750 pc=0x4219e5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000ae7e8 sp=0xc0000ae7e0 pc=0x4744a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x3f2f780?, 0x3?, 0x1e?, 0x86?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000af750 sp=0xc0000af730 pc=0x440dce
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000af7e0 sp=0xc0000af750 pc=0x4219e5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000af7e8 sp=0xc0000af7e0 pc=0x4744a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x3f2f780?, 0x3?, 0xba?, 0x45?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000aff50 sp=0xc0000aff30 pc=0x440dce
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000affe0 sp=0xc0000aff50 pc=0x4219e5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000affe8 sp=0xc0000affe0 pc=0x4744a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x36f76a2d4a384?, 0x1?, 0x50?, 0x2d?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000b0750 sp=0xc0000b0730 pc=0x440dce
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000b07e0 sp=0xc0000b0750 pc=0x4219e5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000b07e8 sp=0xc0000b07e0 pc=0x4744a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 23 [GC worker (idle), 1 minutes]:
runtime.gopark(0x36f61b6f6ce7f?, 0x1?, 0x70?, 0x39?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000b0f50 sp=0xc0000b0f30 pc=0x440dce
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000b0fe0 sp=0xc0000b0f50 pc=0x4219e5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000b0fe8 sp=0xc0000b0fe0 pc=0x4744a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 8 [select, 1 minutes]:
runtime.gopark(0xc000434788?, 0x3?, 0x3?, 0x0?, 0xc000434772?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000434618 sp=0xc0004345f8 pc=0x440dce
runtime.selectgo(0xc000434788, 0xc00043476c, 0xc0005edf80?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000434738 sp=0xc000434618 pc=0x4512a5
go.opencensus.io/stats/view.(*worker).start(0xc0005edf80)
/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f fp=0xc0004347c8 sp=0xc000434738 pc=0xb05e9f
go.opencensus.io/stats/view.init.0.func1()
/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x25 fp=0xc0004347e0 sp=0xc0004347c8 pc=0xb051c5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004347e8 sp=0xc0004347e0 pc=0x4744a1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 278021 [select, 2 minutes]:
runtime.gopark(0xc00231b770?, 0x2?, 0xd8?, 0xb5?, 0xc00231b71c?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00231b598 sp=0xc00231b578 pc=0x440dce
runtime.selectgo(0xc00231b770, 0xc00231b718, 0xc00173e960?, 0x0, 0xc001ae9500?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00231b6b8 sp=0xc00231b598 pc=0x4512a5
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc002e96000)
/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1661 +0x151 fp=0xc00231b7c8 sp=0xc00231b6b8 pc=0x8995f1
google.golang.org/grpc/internal/transport.newHTTP2Client.func10()
/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:388 +0x25 fp=0xc00231b7e0 sp=0xc00231b7c8 pc=0x88f005
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00231b7e8 sp=0xc00231b7e0 pc=0x4744a1
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 278050
/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:388 +0x1e4b

goroutine 112 [select, 850 minutes]:
runtime.gopark(0xc001241f80?, 0x2?, 0x0?, 0x0?, 0xc001241f34?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc001241dc8 sp=0xc001241da8 pc=0x440dce
runtime.selectgo(0xc001241f80, 0xc001241f30, 0x0?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc001241ee8 sp=0xc001241dc8 pc=0x4512a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000e02d80, {0x27859e0, 0xc000e08aa0})
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115 fp=0xc001241fb8 sp=0xc001241ee8 pc=0x7b60d5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.func1()
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc001241fe0 sp=0xc001241fb8 pc=0x7b5f28
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc001241fe8 sp=0xc001241fe0 pc=0x4744a1
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 31 [select, 850 minutes]:
runtime.gopark(0xc000434f80?, 0x2?, 0x2?, 0x0?, 0xc000434f34?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000434dc8 sp=0xc000434da8 pc=0x440dce
runtime.selectgo(0xc000434f80, 0xc000434f30, 0xc0005af2c0?, 0x0, 0x10000c000c33060?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000434ee8 sp=0xc000434dc8 pc=0x4512a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00035d370, {0x27859e0, 0xc0009ce460})
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115 fp=0xc000434fb8 sp=0xc000434ee8 pc=0x7b60d5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.func1()
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc000434fe0 sp=0xc000434fb8 pc=0x7b5f28
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000434fe8 sp=0xc000434fe0 pc=0x4744a1
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 236484 [select, 98 minutes]:
runtime.gopark(0xc00153e780?, 0x2?, 0xe0?, 0x40?, 0xc00153e734?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00153e5c8 sp=0xc00153e5a8 pc=0x440dce
runtime.selectgo(0xc00153e780, 0xc00153e730, 0xc00665d1e0?, 0x0, 0xc00153e788?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00153e6e8 sp=0xc00153e5c8 pc=0x4512a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc006886cd0, {0x27859e0, 0xc001961ef0})
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115 fp=0xc00153e7b8 sp=0xc00153e6e8 pc=0x7b60d5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.func1()
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc00153e7e0 sp=0xc00153e7b8 pc=0x7b5f28
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00153e7e8 sp=0xc00153e7e0 pc=0x4744a1
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 236482
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 23285 [select, 820 minutes]:
runtime.gopark(0xc00480af80?, 0x2?, 0x0?, 0xae?, 0xc00480af34?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00480adc8 sp=0xc00480ada8 pc=0x440dce
runtime.selectgo(0xc00480af80, 0xc00480af30, 0xc005519ff0?, 0x0, 0x7b77a7?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00480aee8 sp=0xc00480adc8 pc=0x4512a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc007b42270, {0x27859e0, 0xc007bee460})
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115 fp=0xc00480afb8 sp=0xc00480aee8 pc=0x7b60d5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.func1()
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc00480afe0 sp=0xc00480afb8 pc=0x7b5f28
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00480afe8 sp=0xc00480afe0 pc=0x4744a1
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 23283
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 23312 [select, 820 minutes]:
runtime.gopark(0xc001219780?, 0x2?, 0x40?, 0xbc?, 0xc001219734?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0012195c8 sp=0xc0012195a8 pc=0x440dce
runtime.selectgo(0xc001219780, 0xc001219730, 0x474000?, 0x0, 0xc001219788?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0012196e8 sp=0xc0012195c8 pc=0x4512a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc008278d00, {0x27859e0, 0xc007a7ee10})
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115 fp=0xc0012197b8 sp=0xc0012196e8 pc=0x7b60d5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.func1()
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc0012197e0 sp=0xc0012197b8 pc=0x7b5f28
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0012197e8 sp=0xc0012197e0 pc=0x4744a1
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 23310
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 51 [select, 850 minutes]:
runtime.gopark(0xc000433f80?, 0x2?, 0x60?, 0xc0?, 0xc000433f34?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000433dc8 sp=0xc000433da8 pc=0x440dce
runtime.selectgo(0xc000433f80, 0xc000433f30, 0x40b580?, 0x0, 0xc000433f38?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000433ee8 sp=0xc000433dc8 pc=0x4512a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00053e440, {0x27859e0, 0xc0009ce6e0})
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115 fp=0xc000433fb8 sp=0xc000433ee8 pc=0x7b60d5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.func1()
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc000433fe0 sp=0xc000433fb8 pc=0x7b5f28
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000433fe8 sp=0xc000433fe0 pc=0x4744a1
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 52 [select, 7 minutes]:
runtime.gopark(0xc000c82f80?, 0x2?, 0xe0?, 0xc1?, 0xc000c82f34?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000c82dc8 sp=0xc000c82da8 pc=0x440dce
runtime.selectgo(0xc000c82f80, 0xc000c82f30, 0xc00043fc70?, 0x0, 0x2?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000c82ee8 sp=0xc000c82dc8 pc=0x4512a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00053e470, {0x27859e0, 0xc0009ce730})
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115 fp=0xc000c82fb8 sp=0xc000c82ee8 pc=0x7b60d5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.func1()
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc000c82fe0 sp=0xc000c82fb8 pc=0x7b5f28
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000c82fe8 sp=0xc000c82fe0 pc=0x4744a1
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 53 [select, 7 minutes]:
runtime.gopark(0xc000c80f80?, 0x2?, 0x0?, 0x0?, 0xc000c80f34?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000c80dc8 sp=0xc000c80da8 pc=0x440dce
runtime.selectgo(0xc000c80f80, 0xc000c80f30, 0x0?, 0x0, 0x1?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000c80ee8 sp=0xc000c80dc8 pc=0x4512a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00053e4a0, {0x27859e0, 0xc0009ce780})
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115 fp=0xc000c80fb8 sp=0xc000c80ee8 pc=0x7b60d5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.func1()
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc000c80fe0 sp=0xc000c80fb8 pc=0x7b5f28
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000c80fe8 sp=0xc000c80fe0 pc=0x4744a1
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 277793 [IO wait]:
runtime.gopark(0x321cdd3339fd8ffc?, 0xb?, 0x0?, 0x0?, 0x47?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0030735c0 sp=0xc0030735a0 pc=0x440dce
runtime.netpollblock(0x489898?, 0x409266?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0030735f8 sp=0xc0030735c0 pc=0x439577
internal/poll.runtime_pollWait(0x7f858fd10408, 0x72)
/usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc003073618 sp=0xc0030735f8 pc=0x46e5e5
internal/poll.(*pollDesc).wait(0xc0031c4700?, 0xc004cc6000?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc003073640 sp=0xc003073618 pc=0x4e5aa7
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0031c4700, {0xc004cc6000, 0x1500, 0x1500})
/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc0030736d8 sp=0xc003073640 pc=0x4e6d9a
net.(*netFD).Read(0xc0031c4700, {0xc004cc6000?, 0xc004cc6005?, 0x22?})
/usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc003073720 sp=0xc0030736d8 pc=0x693a85
net.(*conn).Read(0xc000f07de8, {0xc004cc6000?, 0x3f27be8?, 0xc00257ac38?})
/usr/local/go/src/net/net.go:179 +0x45 fp=0xc003073768 sp=0xc003073720 pc=0x6a4bc5
net.(*TCPConn).Read(0xc003073800?, {0xc004cc6000?, 0xc006117338?, 0x18?})
:1 +0x25 fp=0xc003073798 sp=0xc003073768 pc=0x6b7785
crypto/tls.(*atLeastReader).Read(0xc006117338, {0xc004cc6000?, 0xc006117338?, 0xc000603a00?})
/usr/local/go/src/crypto/tls/conn.go:805 +0x3b fp=0xc0030737e0 sp=0xc003073798 pc=0x76665b
bytes.(*Buffer).ReadFrom(0xc00257ad28, {0x2766a40, 0xc006117338})
/usr/local/go/src/bytes/buffer.go:211 +0x98 fp=0xc003073838 sp=0xc0030737e0 pc=0x5180d8
crypto/tls.(*Conn).readFromUntil(0xc00257aa80, {0x274ef60?, 0xc000f07de8}, 0x1500?)
/usr/local/go/src/crypto/tls/conn.go:827 +0xde fp=0xc003073878 sp=0xc003073838 pc=0x76683e
crypto/tls.(*Conn).readRecordOrCCS(0xc00257aa80, 0x0)
/usr/local/go/src/crypto/tls/conn.go:625 +0x250 fp=0xc003073c18 sp=0xc003073878 pc=0x763e10
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc00257aa80, {0xc001ff8000, 0x8000, 0xc003073cc8?})
/usr/local/go/src/crypto/tls/conn.go:1369 +0x158 fp=0xc003073c88 sp=0xc003073c18 pc=0x76a0d8
google.golang.org/grpc/internal/credentials.(*syscallConn).Read(0x800010601?, {0xc001ff8000?, 0xc003073d30?, 0x800010601?})
:1 +0x29 fp=0xc003073cb8 sp=0xc003073c88 pc=0x7a38a9
bufio.(*Reader).Read(0xc005533440, {0xc003312580, 0x9, 0x7f85dd248f18?})
/usr/local/go/src/bufio/bufio.go:244 +0x197 fp=0xc003073cf0 sp=0xc003073cb8 pc=0x57c5d7
io.ReadAtLeast({0x274c700, 0xc005533440}, {0xc003312580, 0x9, 0x9}, 0x9)
/usr/local/go/src/io/io.go:335 +0x90 fp=0xc003073d38 sp=0xc003073cf0 pc=0x4e0070
io.ReadFull(...)
/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc003312580, 0x9, 0x3edbfc0?}, {0x274c700?, 0xc005533440?})
/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x65 fp=0xc003073d88 sp=0xc003073d38 pc=0x861c05
golang.org/x/net/http2.(*Framer).ReadFrame(0xc003312540)
/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x85 fp=0xc003073e30 sp=0xc003073d88 pc=0x862345
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc003b30d80, 0xc002d987b8?)
/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1587 +0x245 fp=0xc003073fc0 sp=0xc003073e30 pc=0x898e25
google.golang.org/grpc/internal/transport.newHTTP2Client.func11()
/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:398 +0x25 fp=0xc003073fe0 sp=0xc003073fc0 pc=0x88efa5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc003073fe8 sp=0xc003073fe0 pc=0x4744a1
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 277709
/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:398 +0x1ebe

...

What did you expect to see?

i want to know why this error occurred

I built with the following build options: go build -o server -ldflags "-X main.ServiceVersion=$(VERSIONS) -linkmode external"
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Mar 7, 2024
@panjf2000 panjf2000 added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 7, 2024
@panjf2000 panjf2000 added this to the Backlog milestone Mar 7, 2024
@mauri870 mauri870 added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 7, 2024
@mauri870
Copy link
Member

mauri870 commented Mar 7, 2024

Can you share a reproducible example that triggers this issue? Also make sure to run your program with -race to check for race conditions in your code.

@ben-swit
Copy link
Author

ben-swit commented Mar 8, 2024

@mauri870 It occurs intermittently in the operating environment and is difficult to reproduce. There is a reason for a normal panic situation, but could dynamic links be the reason?

@panjf2000
Copy link
Member

@mauri870 It occurs intermittently in the operating environment and is difficult to reproduce. There is a reason for a normal panic situation, but could dynamic links be the reason?

Then did this occur under some specific circumstances? Some additional contexts might help the diagnosis.

@ben-swit
Copy link
Author

ben-swit commented Mar 8, 2024 •
edited
Loading

@panjf2000 I'll make sure to attach it when certain times and situations are checked. Judging from the stack trace, it seems to be the moment of GC execution. Is that correct? Also, I'm attaching the ldd results of the program. We're using Dynatrace, and I suspect it might be a Dynatrace issue.

/lib/ld-musl-x86_64.so.1 (0x7fcb81b18000)
	/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so => /opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so (0x7fcb818f8000)
	libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fcb81b18000)

@ben-swit
Copy link
Author

in my case this is relate of dynatrace so close this issue

@github-project-automation github-project-automation bot moved this from Todo to Done in Go Compiler / Runtime Mar 22, 2024
@gabyhelp gabyhelp mentioned this issue Sep 26, 2024
Closed
This was referenced Oct 9, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Go Compiler / Runtime
Status: Done
Milestone
Backlog
Development

No branches or pull requests
4 participants
@panjf2000 @gopherbot @mauri870 @ben-swit