runtime: crash on linux/arm64 with "invalid p state" on 1.13

[choury]

What version of Go are you using (go version)?

$ go version
go version go1.13.10 linux/arm64

Does this issue reproduce with the latest release?

Not happened with 1.14, but also in 1.13.11

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/data/choury/.cache/go-build"
GOENV="/data/choury/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/data/choury/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/data/choury/.local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/data/choury/.local/go/pkg/tool/linux_arm64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build335762780=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run this code

package main

import (
        "io/ioutil"
        "math/rand"
        "os/exec"
        "time"
)

func main() {
        for i := 0; i < 100; i++ {
                go func() {
                        for {
                                d := time.Duration(rand.Intn(100)+1) * time.Millisecond
                                time.Sleep(d)
                                cmd := exec.Command("ls")
                                cmd.CombinedOutput()
                        }
                }()
                go func() {
                        for {
                                d := time.Duration(rand.Intn(100)+1) * time.Millisecond
                                time.Sleep(d)
                                ioutil.ReadFile("/proc/cpuinfo")
                        }
                }()
        }
        select {}
}

What did you expect to see?

No error, run forever...

What did you see instead?

crash with:

releasep: m=0x4000f82000 m->p=0x400003a500 p->m=0x0 m->mcache=0xfd4da32dac00 p->mcache=0xfd4da32dac00 p->status=0
fatal error: releasep: invalid p state
runtime: newstack at os/exec.(*Cmd).Run+0x78 sp=0x4000baef60 stack=[0x4000bae000, 0x4000baf000]
        morebuf={pc:0x88f2c sp:0x4000baef60 lr:0x0}
        sched={pc:0x883a8 sp:0x4000baef60 lr:0x88f2c ctxt:0x0}
os/exec.(*Cmd).CombinedOutput(0x4000934b00, 0x2, 0x0, 0x0, 0x0, 0x4000934b00)
        /usr/local/go/src/os/exec/exec.go:566 +0x8c fp=0x4000baef90 sp=0x4000baef60 pc=0x88f2c
main.main.func1()
        /crash/main.go:17 +0x80 fp=0x4000baefd0 sp=0x4000baef90 pc=0x8a0a0
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm64.s:1128 +0x4 fp=0x4000baefd0 sp=0x4000baefd0 pc=0x60494
created by main.main
        /crash/main.go:12 +0x3c
fatal error: runtime: stack split at bad time
signal: segmentation fault (core dumped)

[andybons]

If I’m understanding correctly, this reproduces reliably on 1.13, but not 1.14? Is that right?

[andybons]

@aclements

[choury]

If I’m understanding correctly, this reproduces reliably on 1.13, but not 1.14? Is that right?

Yes

[shawndx]

@choury I tried the case on various arm64 machines with 'go1.13.11' but it didn't crash after 10min or so, how long should I expect the error to happen? Once ran into the same error with 'go tool dist test' but never reproduced it again, so want to try with your case. Thanks.

[choury]

@choury I tried the case on various arm64 machines with 'go1.13.11' but it didn't crash after 10min or so, how long should I expect the error to happen? Once ran into the same error with 'go tool dist test' but never reproduced it again, so want to try with your case. Thanks.

go tool dist test:

##### Testing packages.
ok  	archive/tar	0.189s
ok  	archive/zip	0.508s
ok  	bufio	0.632s
ok  	bytes	0.697s
ok  	compress/bzip2	0.087s
ok  	compress/flate	0.677s
ok  	compress/gzip	1.007s
ok  	compress/lzw	0.517s
ok  	compress/zlib	0.560s
ok  	container/heap	0.026s
ok  	container/list	0.507s
ok  	container/ring	0.570s
ok  	context	0.988s
ok  	crypto	0.554s
ok  	crypto/aes	0.586s
ok  	crypto/cipher	0.552s
ok  	crypto/des	0.574s
ok  	crypto/dsa	0.027s
ok  	crypto/ecdsa	0.160s
ok  	crypto/ed25519	0.067s
ok  	crypto/elliptic	0.593s
ok  	crypto/hmac	0.588s
ok  	crypto/internal/subtle	0.583s
ok  	crypto/md5	0.523s
ok  	crypto/rand	0.063s
ok  	crypto/rc4	1.188s
ok  	crypto/rsa	0.591s
ok  	crypto/sha1	0.054s
ok  	crypto/sha256	0.550s
ok  	crypto/sha512	0.470s
ok  	crypto/subtle	0.571s
ok  	crypto/tls	0.776s
ok  	crypto/x509	0.423s
ok  	database/sql	0.588s
ok  	database/sql/driver	0.577s
ok  	debug/dwarf	1.128s
ok  	debug/elf	0.149s
ok  	debug/gosym	0.507s
ok  	debug/macho	0.077s
ok  	debug/pe	0.564s
ok  	debug/plan9obj	0.016s
ok  	encoding/ascii85	0.565s
ok  	encoding/asn1	0.575s
ok  	encoding/base32	0.542s
ok  	encoding/base64	0.549s
ok  	encoding/binary	0.547s
ok  	encoding/csv	0.543s
ok  	encoding/gob	0.062s
ok  	encoding/hex	1.123s
ok  	encoding/json	0.085s
ok  	encoding/pem	0.046s
ok  	encoding/xml	0.543s
ok  	errors	0.545s
ok  	expvar	0.016s
ok  	flag	0.026s
ok  	fmt	0.570s
ok  	go/ast	0.263s
ok  	go/build	0.143s
ok  	go/constant	0.554s
ok  	go/doc	0.600s
ok  	go/format	0.020s
ok  	go/importer	0.371s
ok  	go/internal/gccgoimporter	0.038s
ok  	go/internal/gcimporter	0.606s
ok  	go/internal/srcimporter	1.178s
ok  	go/parser	0.615s
ok  	go/printer	0.856s
ok  	go/scanner	0.543s
ok  	go/token	0.233s
ok  	go/types	1.016s
ok  	hash	0.520s
ok  	hash/adler32	0.563s
ok  	hash/crc32	0.534s
ok  	hash/crc64	0.072s
ok  	hash/fnv	0.019s
ok  	html	0.051s
ok  	html/template	0.527s
ok  	image	0.112s
ok  	image/color	0.581s
ok  	image/draw	0.551s
ok  	image/gif	0.865s
ok  	image/jpeg	0.733s
ok  	image/png	0.550s
ok  	index/suffixarray	0.702s
ok  	internal/cpu	0.146s
ok  	internal/fmtsort	0.558s
ok  	internal/poll	0.077s
ok  	internal/reflectlite	0.554s
ok  	internal/singleflight	0.581s
ok  	internal/trace	0.621s
ok  	internal/xcoff	0.560s
ok  	io	0.635s
ok  	io/ioutil	0.593s
ok  	log	0.509s
ok  	log/syslog	1.251s
ok  	math	0.521s
ok  	math/big	0.840s
ok  	math/bits	0.208s
ok  	math/cmplx	0.536s
ok  	math/rand	0.568s
ok  	mime	0.515s
ok  	mime/multipart	0.194s
ok  	mime/quotedprintable	0.298s
ok  	net	2.931s
ok  	net/http	2.843s
ok  	net/http/cgi	0.253s
ok  	net/http/cookiejar	0.023s
ok  	net/http/fcgi	0.036s
ok  	net/http/httptest	0.040s
ok  	net/http/httptrace	0.016s
ok  	net/http/httputil	0.061s
ok  	net/http/internal	0.575s
ok  	net/http/pprof	2.021s
ok  	net/internal/socktest	0.591s
ok  	net/mail	0.025s
ok  	net/rpc	0.036s
ok  	net/rpc/jsonrpc	0.023s
ok  	net/smtp	0.050s
ok  	net/textproto	0.158s
ok  	net/url	0.107s
ok  	os	2.131s
ok  	os/exec	0.593s
ok  	os/signal	4.676s
ok  	os/user	0.145s
ok  	path	0.023s
ok  	path/filepath	0.526s
ok  	reflect	0.745s
ok  	regexp	0.646s
ok  	regexp/syntax	0.866s
ok  	runtime	14.929s
ok  	runtime/debug	0.071s
ok  	runtime/internal/atomic	0.530s
ok  	runtime/internal/math	0.040s
ok  	runtime/internal/sys	0.043s
ok  	runtime/pprof	5.884s
ok  	runtime/pprof/internal/profile	0.548s
ok  	runtime/trace	0.715s
ok  	sort	0.534s
ok  	strconv	1.010s
ok  	strings	0.578s
ok  	sync	0.853s
ok  	sync/atomic	0.687s
ok  	syscall	0.073s
ok  	testing	0.658s
ok  	testing/quick	0.573s
ok  	text/scanner	0.521s
ok  	text/tabwriter	0.514s
ok  	text/template	0.502s
ok  	text/template/parse	0.470s
ok  	time	3.062s
ok  	unicode	0.465s
ok  	unicode/utf16	0.524s
ok  	unicode/utf8	0.475s
ok  	cmd/addr2line	1.679s
ok  	cmd/api	0.543s
ok  	cmd/asm/internal/asm	0.690s
ok  	cmd/asm/internal/lex	0.032s
ok  	cmd/compile	0.016s
ok  	cmd/compile/internal/gc	6.612s
ok  	cmd/compile/internal/ssa	0.763s
ok  	cmd/compile/internal/syntax	0.530s
ok  	cmd/compile/internal/test	0.536s [no tests to run]
ok  	cmd/compile/internal/types	0.260s
ok  	cmd/cover	1.974s
ok  	cmd/doc	0.332s
ok  	cmd/fix	0.620s
ok  	cmd/go	33.515s
ok  	cmd/go/internal/auth	0.014s
ok  	cmd/go/internal/cache	0.623s
ok  	cmd/go/internal/dirhash	0.463s
ok  	cmd/go/internal/generate	0.045s
ok  	cmd/go/internal/get	0.035s
ok  	cmd/go/internal/imports	0.229s
ok  	cmd/go/internal/load	0.119s
ok  	cmd/go/internal/lockedfile	0.217s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.311s
ok  	cmd/go/internal/modconv	0.027s
ok  	cmd/go/internal/modfetch	0.022s
ok  	cmd/go/internal/modfetch/codehost	0.243s
ok  	cmd/go/internal/modfile	0.335s
ok  	cmd/go/internal/modload	0.022s
ok  	cmd/go/internal/module	0.322s
ok  	cmd/go/internal/mvs	0.536s
ok  	cmd/go/internal/note	0.238s
ok  	cmd/go/internal/par	0.331s
ok  	cmd/go/internal/renameio	0.311s
ok  	cmd/go/internal/search	0.221s
ok  	cmd/go/internal/semver	0.428s
ok  	cmd/go/internal/sumweb	0.053s
ok  	cmd/go/internal/tlog	0.331s
ok  	cmd/go/internal/txtar	0.274s
ok  	cmd/go/internal/web	0.031s
ok  	cmd/go/internal/work	0.113s
ok  	cmd/gofmt	0.251s
ok  	cmd/internal/buildid	0.370s
ok  	cmd/internal/dwarf	0.061s
ok  	cmd/internal/edit	0.169s
ok  	cmd/internal/goobj	0.891s
ok  	cmd/internal/obj	0.028s
ok  	cmd/internal/obj/arm64	0.221s
ok  	cmd/internal/obj/x86	1.875s
ok  	cmd/internal/objabi	0.039s
ok  	cmd/internal/src	0.140s
ok  	cmd/internal/test2json	0.234s
ok  	cmd/link	6.006s
ok  	cmd/link/internal/ld	0.758s
ok  	cmd/link/internal/sym	0.091s
ok  	cmd/nm	2.166s
ok  	cmd/objdump	1.833s
ok  	cmd/pack	1.278s
ok  	cmd/trace	0.043s
ok  	cmd/vet	3.957s

##### os/user with tag osusergo
ok  	os/user	0.003s

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
ok  	runtime	4.667s

##### cmd/go terminal test
skipping terminal test; stdout/stderr not terminals

##### Testing without libgcc.
ok  	crypto/x509	0.325s
ok  	net	0.011s
ok  	os/user	0.005s

##### internal linking of -buildmode=pie
ok  	reflect	1.488s
ok  	os/user	0.337s

##### sync -cpu=10
ok  	sync	0.277s

##### Testing race detector
ok  	runtime/race	10.073s
ok  	flag	1.016s
ok  	net	1.082s
ok  	os	1.093s
ok  	os/exec	3.042s
ok  	encoding/gob	1.018s
ok  	flag	1.016s
ok  	os/exec	3.045s

##### ../misc/cgo/stdio
PASS

##### ../misc/cgo/life
PASS

##### ../misc/cgo/test
PASS
ok  	misc/cgo/test	3.219s
PASS
ok  	misc/cgo/test	3.107s
PASS
ok  	misc/cgo/test	3.305s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/nocgo	0.004s
PASS
ok  	misc/cgo/nocgo	0.005s
PASS
ok  	misc/cgo/nocgo	0.005s
PASS
ok  	misc/cgo/test	3.081s
PASS
ok  	misc/cgo/test	3.257s
PASS
ok  	misc/cgo/test	3.063s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/nocgo	0.006s

##### ../misc/cgo/testgodefs

##### ../misc/cgo/testso
ok  	misc/cgo/testso	0.820s

##### ../misc/cgo/testsovar
ok  	misc/cgo/testsovar	0.904s

##### ../misc/cgo/testcshared
SKIP - short mode and $GO_BUILDER_NAME not set

##### ../misc/cgo/testshared
ok  	misc/cgo/testshared	31.782s

##### ../misc/cgo/testsanitizers
PASS

##### ../misc/cgo/errors
PASS

##### ../misc/cgo/testsigfwd

##### ../test/bench/go1

##### ../test

##### API check
Go version is "go1.13.11", ignoring -next /usr/local/go/api/next.txt

ALL TESTS PASSED

It will crash within 10 min in docker golang:1.13.11 image on my machine.

[shawndx]

@choury Thanks, I verified the case in golang:1.13.11 containers on two different models of machine, it didn't crash after dozens of minutes. Will update later if I'm able to reproduce the issue.

[shawndx]

crash.100.goroutines.txt
crash.1000.Goroutines.txt
crash.10000.Goroutines.abridged.txt
crash.GOTRACEBACK=crash.txt

The case from @choury produced random, and different, crashes with go 1.13.12 (the # of user goroutines ranges between 100 * 2 - 10000 * 2), enclosing several log files.
@cherrymui @ianlancetaylor @andybons @aclements , could you please help to take a look?

[shawndx]

wondering if it's related to #32912, whose fix is not in 1.13.12, trying to verify.

[shawndx]

After cherry-picking the following CLs to 1.13.12 we didn't observe any crash with multiple rounds of testing against the sample case, each running 24 hours at least.
Further verifying in production environment is ongoing and no issue was observed in the past week.

https://go-review.googlesource.com/c/go/+/192937
https://go-review.googlesource.com/c/go/+/204519
https://go-review.googlesource.com/c/go/+/189418

It's still a WA fix and the root cause is not clear to us. So far, the issue is specific to an in-house machine only, here are more details:

After applying CL #192937 and #204519, the sample case still crashed randomly but was due to nil 'm' of the current 'g' all the time, then with CL #189418 the issue was gone.

Typical stack traces of the nil 'm' issue:

3218            msigrestore(getg().m.sigmask)
Loading Go Runtime support.
(gdb) bt
#0  0x0000000000040a34 in syscall.runtime_AfterForkInChild () at /opt/tools/installed/go/src/runtime/proc.go:3218
#1  0x0000000000070760 in syscall.forkAndExecInChild1 (argv0=0x4000ca60f0 "/usr/bin/ls", argv=<error reading variable: access outside bounds of object referenced via synthetic pointer>,
    envv=<error reading variable: access outside bounds of object referenced via synthetic pointer>, chroot=0x0, dir=0x0, attr=0x40002a4d40, sys=0x156a20 <syscall.zeroSysProcAttr>, pipe=97, r1=0, err1=0, p=..., locked=136)
    at /opt/tools/installed/go/src/syscall/exec_linux.go:225
#2  0x0000000000070390 in syscall.forkAndExecInChild (argv0=0x4000ca60f0 "/usr/bin/ls", argv= []*uint8 = {...}, envv= []*uint8 = {...}, chroot=0x0, dir=0x0, attr=0x40002a4d40, sys=0x156a20 <syscall.zeroSysProcAttr>,
    pipe=97, pid=<optimized out>, err=<optimized out>) at /opt/tools/installed/go/src/syscall/exec_linux.go:72
#3  0x00000000000722f0 in syscall.forkExec (argv0="/usr/bin/ls", argv= []string = {...}, attr=0x40002a4d40, pid=<optimized out>, err=...) at /opt/tools/installed/go/src/syscall/exec_unix.go:201
#4  0x000000000007de84 in syscall.StartProcess (argv0="/usr/bin/ls", argv= []string = {...}, pid=0, err=..., attr=<optimized out>, handle=<optimized out>) at /opt/tools/installed/go/src/syscall/exec_unix.go:248
#5  os.startProcess (name="/usr/bin/ls", argv= []string = {...}, attr=0x40002a4ee0, p=0x0, err=...) at /opt/tools/installed/go/src/os/exec_posix.go:52
#6  0x000000000007dbb4 in os.StartProcess (name="/usr/bin/ls", argv= []string = {...}, attr=0x40002a4ee0, ~r3=<optimized out>, ~r4=...) at /opt/tools/installed/go/src/os/exec.go:102
#7  0x0000000000088884 in os/exec.(*Cmd).Start (c=0x40002ac9a0, ~r0=...) at /opt/tools/installed/go/src/os/exec/exec.go:421
#8  0x0000000000088468 in os/exec.(*Cmd).Run (c=0x40002ac9a0, ~r0=...) at /opt/tools/installed/go/src/os/exec/exec.go:339
#9  0x000000000008903c in os/exec.(*Cmd).CombinedOutput (c=0x40002ac9a0, ~r0=..., ~r1=...) at /opt/tools/installed/go/src/os/exec/exec.go:566
#10 0x000000000008a1b0 in main.main.func1 () at /root/crash.go:17
#11 0x0000000000060554 in runtime.goexit () at /opt/tools/installed/go/src/runtime/asm_arm64.s:1128

Dump of assembler code for function syscall.runtime_AfterForkInChild:
   0x0000000000040a10 <+0>:     str     x30, [sp,#-64]!
   0x0000000000040a14 <+4>:     str     x29, [sp,#-8]
   0x0000000000040a18 <+8>:     sub     x29, sp, #0x8
   0x0000000000040a1c <+12>:    orr     x0, xzr, #0x1
   0x0000000000040a20 <+16>:    adrp    x27, 0x171000 <runtime.trace+62944>
   0x0000000000040a24 <+20>:    add     x27, x27, #0xb08
   0x0000000000040a28 <+24>:    strb    w0, [x27]
   0x0000000000040a2c <+28>:    bl      0x4ad60 <runtime.clearSignalHandlers>
   0x0000000000040a30 <+32>:    ldr     x0, [x28,#48]
=> 0x0000000000040a34 <+36>:    ldr     x0, [x0,#128]
   0x0000000000040a38 <+40>:    str     x0, [sp,#48]
   0x0000000000040a3c <+44>:    nop
   0x0000000000040a40 <+48>:    orr     x0, xzr, #0x2
   0x0000000000040a44 <+52>:    str     w0, [sp,#8]
   0x0000000000040a48 <+56>:    add     x0, sp, #0x30
   0x0000000000040a4c <+60>:    str     x0, [sp,#16]
   0x0000000000040a50 <+64>:    str     xzr, [sp,#24]
   0x0000000000040a54 <+68>:    orr     x0, xzr, #0x8
   0x0000000000040a58 <+72>:    str     w0, [sp,#32]
   0x0000000000040a5c <+76>:    bl      0x61110 <runtime.rtsigprocmask>
   0x0000000000040a60 <+80>:    adrp    x27, 0x171000 <runtime.trace+62944>
   0x0000000000040a64 <+84>:    add     x27, x27, #0xb08
   0x0000000000040a68 <+88>:    strb    wzr, [x27]
   0x0000000000040a6c <+92>:    ldr     x29, [sp,#-8]
   0x0000000000040a70 <+96>:    ldr     x30, [sp],#64
   0x0000000000040a74 <+100>:   ret
   0x0000000000040a78 <+104>:   .inst   0x00000000 ; undefined
   0x0000000000040a7c <+108>:   .inst   0x00000000 ; undefined
End of assembler dump.
(gdb) x $x28+48
0x40001eb530:   0x0000000000000000

"1346 gcw := &getg().m.p.ptr().gcw" of runtime.gcmarknewobject runs into the issue.

(gdb) bt
#0  runtime.systemstack () at /root/xd/go.hw/src/runtime/asm_arm64.s:193
#1  0x000000000003735c in runtime.throw (s="g.m == nil") at /root/xd/go.hw/src/runtime/panic.go:769
#2  0x000000000004b3d0 in runtime.sigtrampgo (sig=<optimized out>, info=0x4000294da0, ctx=0x4000294e20) at /root/xd/go.hw/src/runtime/signal_unix.go:332
#3  0x0000000000061288 in runtime.sigtramp () at /root/xd/go.hw/src/runtime/sys_linux_arm64.s:396
#4  <signal handler called>
#5  0x000000000002b854 in runtime.gcmarknewobject (obj=274882952224, size=96, scanSize=80) at /root/xd/go.hw/src/runtime/mgcmark.go:1346
#6  0x000000000001a804 in runtime.mallocgc (size=96, typ=0xb6de0, needzero=true, ~r3=<optimized out>) at /root/xd/go.hw/src/runtime/malloc.go:1079
#7  0x000000000001add8 in runtime.newobject (typ=0xb6de0, ~r1=<optimized out>) at /root/xd/go.hw/src/runtime/malloc.go:1151
#8  0x000000000007f06c in os.newFile (fd=273, name="/proc/cpuinfo", kind=1, ~r3=<optimized out>) at /root/xd/go.hw/src/os/file_unix.go:116
#9  0x000000000007f37c in os.openFileNolog (name="/proc/cpuinfo", flag=0, perm=0, ~r3=<optimized out>, ~r4=...) at /root/xd/go.hw/src/os/file_unix.go:222
#10 0x000000000007ef14 in os.OpenFile (name="/proc/cpuinfo", flag=0, perm=0, ~r3=<optimized out>, ~r4=...) at /root/xd/go.hw/src/os/file.go:300
#11 0x00000000000857a8 in os.Open (name="/proc/cpuinfo") at /root/xd/go.hw/src/os/file.go:280
#12 io/ioutil.ReadFile (filename="/proc/cpuinfo", ~r1=..., ~r2=...) at /root/xd/go.hw/src/io/ioutil/ioutil.go:53
#13 0x000000000008a25c in main.main.func2 () at /root/xd/0803/go.hw.2/crash.go:24
#14 0x0000000000060584 in runtime.goexit () at /root/xd/go.hw/src/runtime/asm_arm64.s:1128
(gdb) f 5
#5  0x000000000002b854 in runtime.gcmarknewobject (obj=274882952224, size=96, scanSize=80) at /root/xd/go.hw/src/runtime/mgcmark.go:1346
1346            gcw := &getg().m.p.ptr().gcw

   0x000000000002b84c <+60>:    cbnz    x27, 0x2b840 <runtime.gcmarknewobject+48>
   0x000000000002b850 <+64>:    ldr     x0, [x28,#48]
=> 0x000000000002b854 <+68>:    ldr     x0, [x0,#208]
   0x000000000002b858 <+72>:    ldrsb   x27, [x0]
   0x000000000002b85c <+76>:    ldr     x1, [x0,#4736]
   0x000000000002b860 <+80>:    ldr     x2, [sp,#64]

(gdb) p $x0
$3 = 0
(gdb) p/x $x28
$6 = 0x4000211e00
(gdb) p 'runtime.allgs'.array[196]
$7 = (runtime.g *) 0x4000211e00
(gdb) p/x *'runtime.allgs'.array[196]
$10 = {
  stack = {
    lo = 0x40009c9800,
    hi = 0x40009ca000
  },
  stackguard0 = 0x40009c9b70,
  stackguard1 = 0xffffffffffffffff,
  _panic = 0x0,
  _defer = 0x0,
  m = 0x0,
  sched = {
    sp = 0x40009c9f10,
    pc = 0x38f80,
    g = 0x4000211e00,
    ctxt = 0x0,
    ret = 0x0,
    lr = 0x0,
    bp = 0x40009c9f08
  },
  syscallsp = 0x0,
  syscallpc = 0x73e90,
  stktopsp = 0x40009c9fd0,
  param = 0x0,
  atomicstatus = 0x4,
  stackLock = 0x0,
  goid = 0xfd,
  schedlink = 0x4000157200,
  waitsince = 0x398a05ec59758,
  waitreason = 0x13,
  preempt = 0x0,
  paniconfault = 0x0,
  preemptscan = 0x0,
  gcscandone = 0x1,
  gcscanvalid = 0x1,
  throwsplit = 0x0,
  raceignore = 0x0,
  sysblocktraced = 0x1,
  sysexitticks = 0x0,
  traceseq = 0x0,
  tracelastp = 0x0,
  lockedm = 0x0,
  sig = 0x0,
  writebuf =  []uint8,
  sigcode0 = 0x0,
  sigcode1 = 0x0,
  sigpc = 0x0,
  gopc = 0x8a130,
  ancestors =  []runtime.ancestorInfo *<error reading variable: Cannot access memory at address 0x8>,
  startpc = 0x8a1f0,
  racectx = 0x0,
  waiting = 0x0,
  cgoCtxt =  []uintptr,
  labels = 0x0,
  timer = 0x400013c200,
  selectDone = 0x0,
  gcAssistBytes = 0x0
}

@cherrymui are the stack traces useful for helping narrow down the root cause? I could not make it reproducible on other arm64 machines anyway. Thanks a lot.

[seankhliao]

closing as this only appears to affect 1.13