golang 1.14 related problem(mipsle crash, 100% cpu) #771

wangvisual · 2020-03-22T10:22:44Z

20200201 version works, but for version 20200226 && 20200321, the --help may even crashes, but not always, when it crash, the log shows:
root@localhost:/opt/bin/a# ./client_linux_mipsle.0321 --help
runtime: pipe failed with -89
fatal error: runtime: pipe failed

goroutine 6 [running]:
runtime.throw(0x40afb2, 0x14)
/usr/local/go/src/runtime/panic.go:1114 +0x60 fp=0xc285e8 sp=0xc285d4 pc=0x4cd0c
runtime.netpollinit()
/usr/local/go/src/runtime/netpoll_epoll.go:40 +0x20c fp=0xc28620 sp=0xc285e8 pc=0x489a0
runtime.netpollGenericInit()
/usr/local/go/src/runtime/netpoll.go:112 +0xa0 fp=0xc28628 sp=0xc28620 pc=0x46cf4
runtime.doaddtimer(0xc1e000, 0xc3e444)
/usr/local/go/src/runtime/time.go:271 +0x1e8 fp=0xc28654 sp=0xc28628 pc=0x756a0
runtime.addInitializedTimer(0xc3e444)
/usr/local/go/src/runtime/time.go:258 +0x8c fp=0xc28670 sp=0xc28654 pc=0x75480
runtime.addtimer(0xc3e444)
/usr/local/go/src/runtime/time.go:248 +0x88 fp=0xc2867c sp=0xc28670 pc=0x753c4
time.startTimer(0xc3e444)
/usr/local/go/src/runtime/time.go:207 +0x3c fp=0xc28684 sp=0xc2867c pc=0x751c4
time.NewTimer(0x0, 0x0, 0x0)
/usr/local/go/src/time/sleep.go:93 +0xf8 fp=0xc286a8 sp=0xc28684 pc=0xead00
github.com/xtaci/kcp-go/v5.(*TimedSched).sched(0xc48060)
/home/xtaci/go/pkg/mod/github.com/xtaci/kcp-go/[email protected]/timedsched.go:64 +0x5c fp=0xc287e4 sp=0xc286a8 pc=0x225e50
runtime.goexit()
/usr/local/go/src/runtime/asm_mipsx.s:651 +0x4 fp=0xc287e4 sp=0xc287e4 pc=0x87d64
created by github.com/xtaci/kcp-go/v5.NewTimedSched
/home/xtaci/go/pkg/mod/github.com/xtaci/kcp-go/[email protected]/timedsched.go:56 +0x118

goroutine 1 [runnable, locked to thread]:
text/template/parse.init()
/usr/local/go/src/text/template/parse/lex.go:75

goroutine 7 [runnable]:
github.com/xtaci/kcp-go/v5.(*TimedSched).prepend(0xc48060)
/home/xtaci/go/pkg/mod/github.com/xtaci/kcp-go/[email protected]/timedsched.go:93
created by github.com/xtaci/kcp-go/v5.NewTimedSched
/home/xtaci/go/pkg/mod/github.com/xtaci/kcp-go/[email protected]/timedsched.go:58 +0x154

system type : Broadcom BCM4716 chip rev 1 pkg 10
processor : 0
cpu model : MIPS 74K V4.0
BogoMIPS : 239.20
cpu MHz : 480
wait instruction : no
microsecond timers : yes
tlb_entries : 64
extra interrupt vector : no
hardware watchpoint : yes
ASEs implemented : mips16 dsp
shadow register sets : 1
VCED exceptions : not available
VCEI exceptions : not available

Linux localhost 2.6.22.19 #20 Fri May 5 00:20:55 CEST 2017 mips GNU/Linux

xtaci · 2020-03-22T14:14:36Z

是的，golang1.14得timer行为有问题

xtaci · 2020-03-22T14:20:50Z

我放一个更严格的时钟测试一下： @maxwellzh @pokedo0
kcptun-linux-amd64-20200322.tar.gz
kcptun-linux-arm5-20200322.tar.gz
kcptun-linux-arm6-20200322.tar.gz
kcptun-linux-arm7-20200322.tar.gz
kcptun-linux-arm64-20200322.tar.gz
kcptun-linux-mips-20200322.tar.gz
kcptun-linux-mipsle-20200322.tar.gz
kcptun-windows-386-20200322.tar.gz
kcptun-windows-amd64-20200322.tar.gz

kcptun-linux-386-20200322.tar.gz

kcptun-darwin-amd64-20200322.tar.gz
kcptun-freebsd-amd64-20200322.tar.gz

maxwellzh · 2020-03-22T14:45:53Z

0322在服务器（linux64）和本地（darwin）之间开了一个连接，挂了一晚上没出现0321的CPU100%情况

wangvisual · 2020-03-22T15:06:37Z

还是不行：
root@weiw-e4200:/opt/bin/a# ./client_linux_mipsle.0322 --help
NAME:
kcptun - client(with SMUX)

USAGE:
client_linux_mipsle.0322 [global options] command [command options] [arguments...]

VERSION:
20200322

COMMANDS:
help, h Shows a list of commands or help for one command

GLOBAL OPTIONS:
--localaddr value, -l value local listen address (default: ":12948")
--remoteaddr value, -r value kcp server address (default: "vps:29900")
--key value pre-shared secret between client and server (default: "it's a secrect") [$KCPTUN_KEY]
--crypt value aes, aes-128, aes-192, salsa20, blowfish, twofish, cast5, 3des, tea, xtea, xor, sm4, none (default: "aes")
--mode value profiles: fast3, fast2, fast, normal, manual (default: "fast")
--conn value runtime: pipe failed with -89
fatal error: runtime: pipe failed

goroutine 6 [running]:
runtime.throw(0x40afd2, 0x14)
/usr/local/go/src/runtime/panic.go:1114 +0x60 fp=0xc285d0 sp=0xc285bc pc=0x4cd0c
runtime.netpollinit()
/usr/local/go/src/runtime/netpoll_epoll.go:40 +0x20c fp=0xc28608 sp=0xc285d0 pc=0x489a0
runtime.netpollGenericInit()
/usr/local/go/src/runtime/netpoll.go:112 +0xa0 fp=0xc28610 sp=0xc28608 pc=0x46cf4
runtime.doaddtimer(0xc1e000, 0xc3e884)
/usr/local/go/src/runtime/time.go:271 +0x1e8 fp=0xc2863c sp=0xc28610 pc=0x756a0
runtime.addInitializedTimer(0xc3e884)
/usr/local/go/src/runtime/time.go:258 +0x8c fp=0xc28658 sp=0xc2863c pc=0x75480
runtime.addtimer(0xc3e884)
/usr/local/go/src/runtime/time.go:248 +0x88 fp=0xc28664 sp=0xc28658 pc=0x753c4
time.startTimer(0xc3e884)
/usr/local/go/src/runtime/time.go:207 +0x3c fp=0xc2866c sp=0xc28664 pc=0x751c4
time.NewTimer(0x0, 0x0, 0x0)
/usr/local/go/src/time/sleep.go:93 +0xf8 fp=0xc28690 sp=0xc2866c pc=0xead00
github.com/xtaci/kcp-go/v5.(*TimedSched).sched(0xc48060)
/home/xtaci/go/pkg/mod/github.com/xtaci/kcp-go/[email protected]/timedsched.go:64 +0x5c fp=0xc287e4 sp=0xc28690 pc=0x225f28
runtime.goexit()
/usr/local/go/src/runtime/asm_mipsx.s:651 +0x4 fp=0xc287e4 sp=0xc287e4 pc=0x87d64
created by github.com/xtaci/kcp-go/v5.NewTimedSched
/home/xtaci/go/pkg/mod/github.com/xtaci/kcp-go/[email protected]/timedsched.go:56 +0x118

goroutine 1 [runnable]:
syscall.Syscall(0xfa4, 0x1, 0xc6c434, 0x5, 0x5, 0x2, 0x0)
/usr/local/go/src/syscall/asm_linux_mipsx.s:17 +0x10
syscall.write(0x1, 0xc6c434, 0x5, 0x8, 0x0, 0x0, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_mipsle.go:914 +0x60
syscall.Write(...)
/usr/local/go/src/syscall/syscall_unix.go:214
internal/poll.(*FD).Write(0xc3e080, 0xc6c434, 0x5, 0x8, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:268 +0x174
os.(*File).write(...)
/usr/local/go/src/os/file_unix.go:280
os.(*File).Write(0xc0c0f8, 0xc6c434, 0x5, 0x8, 0x8, 0x0, 0x0)
/usr/local/go/src/os/file.go:153 +0x7c
text/tabwriter.(*Writer).write0(0xc6c420, 0xc6c434, 0x5, 0x8)
/usr/local/go/src/text/tabwriter/tabwriter.go:254 +0x60
text/tabwriter.(*Writer).writeN(0xc6c420, 0xc6c434, 0x8, 0x8, 0xd)
/usr/local/go/src/text/tabwriter/tabwriter.go:268 +0xac
text/tabwriter.(*Writer).writePadding(0xc6c420, 0xf, 0x24, 0x600)
/usr/local/go/src/text/tabwriter/tabwriter.go:293 +0x114
text/tabwriter.(*Writer).writeLines(0xc6c420, 0x0, 0x0, 0x1c, 0xcdcb00)
/usr/local/go/src/text/tabwriter/tabwriter.go:324 +0x30c
text/tabwriter.(*Writer).format(0xc6c420, 0x0, 0x0, 0x1c, 0x0)
/usr/local/go/src/text/tabwriter/tabwriter.go:409 +0x2ec
text/tabwriter.(*Writer).format(0xc6c420, 0x0, 0x0, 0x1d, 0xc53300)
/usr/local/go/src/text/tabwriter/tabwriter.go:403 +0x1f0
text/tabwriter.(*Writer).flushNoDefers(0xc6c420)
/usr/local/go/src/text/tabwriter/tabwriter.go:520 +0xe0
text/tabwriter.(*Writer).flush(0xc6c420, 0x0, 0x0)
/usr/local/go/src/text/tabwriter/tabwriter.go:502 +0x94
text/tabwriter.(*Writer).Flush(...)
/usr/local/go/src/text/tabwriter/tabwriter.go:495
github.com/urfave/cli.printHelpCustom(0x47fa10, 0xc0c0f8, 0x41a1cc, 0x41f, 0x3e4080, 0xc01340, 0x0)
/home/xtaci/go/pkg/mod/github.com/urfave/[email protected]/help.go:258 +0x3ac
github.com/urfave/cli.printHelp(0x47fa10, 0xc0c0f8, 0x41a1cc, 0x41f, 0x3e4080, 0xc01340)
/home/xtaci/go/pkg/mod/github.com/urfave/[email protected]/help.go:262 +0x68
github.com/urfave/cli.ShowAppHelp(0xc86210, 0xc86201, 0x40)
/home/xtaci/go/pkg/mod/github.com/urfave/[email protected]/help.go:143 +0x19c
github.com/urfave/cli.(*App).Run(0xc01340, 0xc0a040, 0x2, 0x2, 0x0, 0x0)
/home/xtaci/go/pkg/mod/github.com/urfave/[email protected]/app.go:222 +0x8e0
main.main()
/home/xtaci/go/src/github.com/xtaci/kcptun/client/main.go:457 +0x11dc

goroutine 7 [runnable]:
github.com/xtaci/kcp-go/v5.(*TimedSched).prepend(0xc48060)
/home/xtaci/go/pkg/mod/github.com/xtaci/kcp-go/[email protected]/timedsched.go:100
created by github.com/xtaci/kcp-go/v5.NewTimedSched
/home/xtaci/go/pkg/mod/github.com/xtaci/kcp-go/[email protected]/timedsched.go:58 +0x154

goroutine 8 [runnable]:
main.sigHandler()
/home/xtaci/go/src/github.com/xtaci/kcptun/client/signal.go:18
created by main.init.0
/home/xtaci/go/src/github.com/xtaci/kcptun/client/signal.go:15 +0x44

xtaci · 2020-03-22T15:29:58Z

@wangvisual 你的这个问题，可否查下dmesg，或者syslog有没有异常，虽然这是 golang的bug，但errno == -89，这个不知道是啥

xtaci · 2020-03-22T15:30:59Z

EDESTADDRREQ 89 Destination address required

wangvisual · 2020-03-22T17:37:48Z

log里没有可疑的信息。

On MIPS, pipe returns two values rather than taking a pointer. On MIPS64, call pipe2 rather than pipe. Also, use the correct system call number for fcntl on mips64. Change-Id: Ie72acdefeb593f44cb98735fc40eac99cf73509e Reviewed-on: https://go-review.googlesource.com/c/go/+/202417 Run-TryBot: Ian Lance Taylor <[email protected]> Reviewed-by: Tobias Klauser <[email protected]> TryBot-Result: Gobot Gobot <[email protected]>

wangvisual · 2020-03-22T18:35:37Z

1.14 has one additional call nonblockingPipe() within netpollinit() compare with 1.13, the pipe() definition for mipsle was not verified.

xtaci · 2020-03-23T04:32:57Z

golang officially confirmed that issue, but there're still some problem with timer on amd64... some time timer hangs.

syuez · 2020-03-23T13:55:18Z

@xtaci 试了20200322的版本，服务端还是会出现CPU占用99.9的情况，只是比20200321版本运行的时间长了，从早上9到21点。

xtaci · 2020-03-23T14:31:33Z

golang/go#38023
@syuez 是的，我追查到的golang的bug

xtaci · 2020-03-24T02:31:08Z

@wangvisual can you help run this build, with patch from golang/go#37997

kcptun-linux-mipsle-20200324.tar.gz

kcptun-linux-mips-20200324.tar.gz

xtaci · 2020-03-24T09:55:29Z

kcptun-darwin-amd64-20200324.tar.gz
kcptun-freebsd-amd64-20200324.tar.gz
kcptun-linux-386-20200324.tar.gz
kcptun-linux-amd64-20200324.tar.gz
kcptun-linux-arm5-20200324.tar.gz
kcptun-linux-arm6-20200324.tar.gz
kcptun-linux-arm7-20200324.tar.gz
kcptun-linux-arm64-20200324.tar.gz
kcptun-linux-mips-20200324.tar.gz
kcptun-linux-mipsle-20200324.tar.gz
kcptun-windows-386-20200324.tar.gz
kcptun-windows-amd64-20200324.tar.gz

@maxwellzh @syuez 我打了golang的patch，看上去是消灭了100% cpu的问题了，可以测试下。

syuez · 2020-03-25T00:49:20Z

@xtaci CPU 100%的问题没有了，但是发现客户端内存好像有泄露问题

从昨天21点运行到早上8点，内存占用了80%左右

未 kill掉进程前内存

运行时内存占用

kill 掉后内存

用的是TCP模式，同机子另外以UDP模式运行的客户端则没有这个问题

maxwellzh · 2020-03-25T03:57:07Z

@xtaci 从昨晚到现在一切正常，没出现CPU满载的情况

xtaci · 2020-03-25T04:09:40Z

@xtaci CPU 100%的问题没有了，但是发现客户端内存好像有泄露问题

从昨天21点运行到早上8点，内存占用了80%左右

未 kill掉进程前内存

运行时内存占用

kill 掉后内存

用的是TCP模式，同机子另外以UDP模式运行的客户端则没有这个问题

TCP模式没有经过特别的优化，可能会有较大的内存消耗，空了再做优化。其他没问题就好，你可以尝试启动的时候加入 GOGC=10 ./client_xxx 这样启动。

xtaci · 2020-03-25T04:10:21Z

@xtaci 从昨晚到现在一切正常，没出现CPU满载的情况

golang可能会在下一个小版本修复此问题。golang/go#38023

syuez · 2020-03-25T04:31:47Z

@xtaci
GOGC 怎么跟 nohup 命令配合？
nohup GOGC=10 /root/client_linux_amd64 -c /root/config.json >/dev/null 2>&1 &
~~我尝试这样的方式但是无法运行~~

用sh脚本套娃好了😂

wangvisual · 2020-03-25T04:41:18Z

@wangvisual can you help run this build, with patch from golang/go#37997

kcptun-linux-mipsle-20200324.tar.gz

kcptun-linux-mips-20200324.tar.gz
可以了，非常感谢。

wangvisual · 2020-03-25T05:13:25Z

@xtaci
~GOGC 怎么跟 nohup 命令配合？ ~nohup GOGC=10 /root/client_linux_amd64 -c /root/config.json >/dev/null 2>&1 & ~我尝试这样的方式但是无法运行~~~

用sh脚本套娃好了😂

nohup env GOGC=10 ...

syuez · 2020-03-25T05:51:14Z

@xtaci
~GOGC 怎么跟 nohup 命令配合？ ~nohup GOGC=10 /root/client_linux_amd64 -c /root/config.json >/dev/null 2>&1 & ~我尝试这样的方式但是无法运行~~~
用sh脚本套娃好了😂

nohup env GOGC=10 ...

感谢！

qx220 · 2020-04-10T12:51:33Z

v20200409的docker版本服务端在运行一段时间之后依旧有卡死的问题（看服务端的日志：网络没问题，服务端无法收到任何packet）

xtaci · 2020-04-10T13:54:59Z

@qx-juilliard docker版本可能没有更新golang版本，我更新了一下Dockerfile，可以pull latest试试

tete1030 · 2020-04-28T03:48:47Z

@xtaci CPU 100%的问题没有了，但是发现客户端内存好像有泄露问题
从昨天21点运行到早上8点，内存占用了80%左右
未 kill掉进程前内存
运行时内存占用
kill 掉后内存
用的是TCP模式，同机子另外以UDP模式运行的客户端则没有这个问题

TCP模式没有经过特别的优化，可能会有较大的内存消耗，空了再做优化。其他没问题就好，你可以尝试启动的时候加入 GOGC=10 ./client_xxx 这样启动。

tcp模式确实存在内存泄漏问题，附图是client进程的内存消耗，准备试试GOGC

syuez · 2020-04-28T09:02:15Z

@xtaci CPU 100%的问题没有了，但是发现客户端内存好像有泄露问题
从昨天21点运行到早上8点，内存占用了80%左右
未 kill掉进程前内存
运行时内存占用
kill 掉后内存
用的是TCP模式，同机子另外以UDP模式运行的客户端则没有这个问题

TCP模式没有经过特别的优化，可能会有较大的内存消耗，空了再做优化。其他没问题就好，你可以尝试启动的时候加入 GOGC=10 ./client_xxx 这样启动。

tcp模式确实存在内存泄漏问题，附图是client进程的内存消耗，准备试试GOGC

目前只有20200226版本好一点，之后的版本基本1天内存就爆了(VPS 512M)，我现在服务器和客户端都换成0226版本了😂

tete1030 · 2020-04-30T08:53:58Z

Not for me. Mine is related to tcp simulation mode. Normal mode has no such problem.

After GOGC=1, the situation seems a little better, but still growing memory.

syuez · 2020-05-01T02:02:13Z

@tete1030 @syuez #677
Is this related to the issue mentioned in 667 ？

If use tcp mode it will cause problem,udp will not.

semigodking · 2020-06-21T14:07:29Z

CPU usage is also high on Raspberry Pi 2.

xtaci mentioned this issue Mar 22, 2020

最新版(20200321)的TCP模式好像有点问题 #769

Closed

xtaci pinned this issue Mar 22, 2020

xtaci mentioned this issue Mar 23, 2020

runtime: "pipe failed with -89" at program startup(mipsle only), timer related netpoll init. golang/go#37997

Closed

xtaci changed the title ~~client_linux_mipsle random crashes when start for latest version(20200321)~~ golang 1.14 related problem(mipsle crash, 100% cpu) Mar 25, 2020

tete1030 mentioned this issue Apr 22, 2020

server端程序疑似挂起 #785

Closed

xtaci unpinned this issue Oct 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

golang 1.14 related problem(mipsle crash, 100% cpu) #771

golang 1.14 related problem(mipsle crash, 100% cpu) #771

wangvisual commented Mar 22, 2020

xtaci commented Mar 22, 2020

xtaci commented Mar 22, 2020

maxwellzh commented Mar 22, 2020 •

edited

Loading

wangvisual commented Mar 22, 2020

xtaci commented Mar 22, 2020

xtaci commented Mar 22, 2020

wangvisual commented Mar 22, 2020

wangvisual commented Mar 22, 2020

xtaci commented Mar 23, 2020

syuez commented Mar 23, 2020

xtaci commented Mar 23, 2020

xtaci commented Mar 24, 2020 •

edited

Loading

xtaci commented Mar 24, 2020

syuez commented Mar 25, 2020

maxwellzh commented Mar 25, 2020

xtaci commented Mar 25, 2020

xtaci commented Mar 25, 2020

syuez commented Mar 25, 2020 •

edited

Loading

wangvisual commented Mar 25, 2020

wangvisual commented Mar 25, 2020

syuez commented Mar 25, 2020

qx220 commented Apr 10, 2020

xtaci commented Apr 10, 2020

tete1030 commented Apr 28, 2020 •

edited

Loading

syuez commented Apr 28, 2020

tete1030 commented Apr 30, 2020

syuez commented May 1, 2020

semigodking commented Jun 21, 2020

golang 1.14 related problem(mipsle crash, 100% cpu) #771

golang 1.14 related problem(mipsle crash, 100% cpu) #771

Comments

wangvisual commented Mar 22, 2020

xtaci commented Mar 22, 2020

xtaci commented Mar 22, 2020

maxwellzh commented Mar 22, 2020 • edited Loading

wangvisual commented Mar 22, 2020

xtaci commented Mar 22, 2020

xtaci commented Mar 22, 2020

wangvisual commented Mar 22, 2020

wangvisual commented Mar 22, 2020

xtaci commented Mar 23, 2020

syuez commented Mar 23, 2020

xtaci commented Mar 23, 2020

xtaci commented Mar 24, 2020 • edited Loading

xtaci commented Mar 24, 2020

syuez commented Mar 25, 2020

maxwellzh commented Mar 25, 2020

xtaci commented Mar 25, 2020

xtaci commented Mar 25, 2020

syuez commented Mar 25, 2020 • edited Loading

wangvisual commented Mar 25, 2020

wangvisual commented Mar 25, 2020

syuez commented Mar 25, 2020

qx220 commented Apr 10, 2020

xtaci commented Apr 10, 2020

tete1030 commented Apr 28, 2020 • edited Loading

syuez commented Apr 28, 2020

tete1030 commented Apr 30, 2020

syuez commented May 1, 2020

semigodking commented Jun 21, 2020

maxwellzh commented Mar 22, 2020 •

edited

Loading

xtaci commented Mar 24, 2020 •

edited

Loading

syuez commented Mar 25, 2020 •

edited

Loading

tete1030 commented Apr 28, 2020 •

edited

Loading