pubsub: weird panic ( pubsub.NewClient ) #229

azr · 2016-03-22T16:53:43Z

Hello there,

I have a service running in GCP kubernetes and right after calling the latest pubsub.NewClient(ctx, *projID) it panics:

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xe5 pc=0x7fe7425dda5c]

runtime stack:
runtime.throw(0x1295d20, 0x2a)
    /usr/local/go/src/runtime/panic.go:530 +0x90
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0xb602b0, 0xc820547310, 0x0)
    /usr/local/go/src/runtime/cgocall.go:123 +0x11b fp=0xc8205472b8 sp=0xc820547288
os/user._Cfunc_mygetpwuid_r(0x0, 0xc8202c41e0, 0x7fe7040008c0, 0x400, 0xc82002c140, 0x0)
    ??:0 +0x44 fp=0xc820547310 sp=0xc8205472b8
os/user.lookupUnix(0x0, 0x0, 0x0, 0x1166c00, 0x0, 0x0, 0x0)
    /usr/local/go/src/os/user/lookup_unix.go:95 +0xcd8 fp=0xc8205474a8 sp=0xc820547310
os/user.current(0x4759a0, 0x0, 0x0)
    /usr/local/go/src/os/user/lookup_unix.go:38 +0x42 fp=0xc8205474e8 sp=0xc8205474a8
os/user.Current(0x54fc0d, 0x0, 0x0)
    /usr/local/go/src/os/user/lookup.go:9 +0x24 fp=0xc820547508 sp=0xc8205474e8
golang.org/x/oauth2/google.guessUnixHomeDir(0x0, 0x0)
    /go/src/xx/xx/Godeps/_workspace/src/golang.org/x/oauth2/google/sdk.go:163 +0x28 fp=0xc820547540 sp=0xc820547508
golang.org/x/oauth2/google.wellKnownFile(0x0, 0x0)
    /go/src/xx/xx/Godeps/_workspace/src/golang.org/x/oauth2/google/default.go:107 +0x36 fp=0xc8205475d8 sp=0xc820547540
golang.org/x/oauth2/google.DefaultTokenSource(0x7fe74318b6b8, 0xc820072d78, 0xc820390300, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0)
    /go/src/xx/xx/Godeps/_workspace/src/golang.org/x/oauth2/google/default.go:72 +0x307 fp=0xc820547780 sp=0xc8205475d8
google.golang.org/cloud/internal/transport.NewHTTPClient(0x7fe74318b6b8, 0xc820072d78, 0xc820547a38, 0x3, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/xx/xx/Godeps/_workspace/src/google.golang.org/cloud/internal/transport/dial.go:64 +0x2c1 fp=0xc8205478a8 sp=0xc820547780
google.golang.org/cloud/pubsub.NewClient(0x7fe74318b6b8, 0xc820072d78, 0x7fff2e3248f2, 0x7, 0x0, 0x0, 0x0, 0xc800000002, 0x0, 0x0)
    /go/src/xx/xx/Godeps/_workspace/src/google.golang.org/cloud/pubsub/pubsub.go:62 +0x45e fp=0xc820547a70 sp=0xc8205478a8
main.main()
    /go/src/xx/xx/main.go:179 +0xc34 fp=0xc820547f10 sp=0xc820547a70
runtime.main()
    /usr/local/go/src/runtime/proc.go:188 +0x2b0 fp=0xc820547f60 sp=0xc820547f10
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820547f68 sp=0xc820547f60

[...]

It works locally if I use a json key file. ( on docker )

Cheers ! :)

The text was updated successfully, but these errors were encountered:

broady · 2016-03-22T16:57:00Z

Makes sense that would fix it. It's trying to find the home directory with
application default credentials.

Can you reproduce this panic consistently? If so, can you tell us more
about how your app is configured and deployed? Do you have the Dockerfile
available?
On Mar 22, 2016 9:53 AM, "Adrien Delorme" [email protected] wrote:

Hello there,

I have a service running in GCP kubernetes and right after calling the
latest pubsub.NewClient(ctx, *projID) it panics:

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xe5 pc=0x7fe7425dda5c]

runtime stack:
runtime.throw(0x1295d20, 0x2a)
/usr/local/go/src/runtime/panic.go:530 +0x90
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0xb602b0, 0xc820547310, 0x0)
/usr/local/go/src/runtime/cgocall.go:123 +0x11b fp=0xc8205472b8 sp=0xc820547288
os/user._Cfunc_mygetpwuid_r(0x0, 0xc8202c41e0, 0x7fe7040008c0, 0x400, 0xc82002c140, 0x0)
??:0 +0x44 fp=0xc820547310 sp=0xc8205472b8
os/user.lookupUnix(0x0, 0x0, 0x0, 0x1166c00, 0x0, 0x0, 0x0)
/usr/local/go/src/os/user/lookup_unix.go:95 +0xcd8 fp=0xc8205474a8 sp=0xc820547310
os/user.current(0x4759a0, 0x0, 0x0)
/usr/local/go/src/os/user/lookup_unix.go:38 +0x42 fp=0xc8205474e8 sp=0xc8205474a8
os/user.Current(0x54fc0d, 0x0, 0x0)
/usr/local/go/src/os/user/lookup.go:9 +0x24 fp=0xc820547508 sp=0xc8205474e8golang.org/x/oauth2/google.guessUnixHomeDir(0x0, 0x0)
/go/src/xx/xx/Godeps/_workspace/src/golang.org/x/oauth2/google/sdk.go:163 +0x28 fp=0xc820547540 sp=0xc820547508golang.org/x/oauth2/google.wellKnownFile(0x0, 0x0)
/go/src/xx/xx/Godeps/_workspace/src/golang.org/x/oauth2/google/default.go:107 +0x36 fp=0xc8205475d8 sp=0xc820547540golang.org/x/oauth2/google.DefaultTokenSource(0x7fe74318b6b8, 0xc820072d78, 0xc820390300, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0)
/go/src/xx/xx/Godeps/_workspace/src/golang.org/x/oauth2/google/default.go:72 +0x307 fp=0xc820547780 sp=0xc8205475d8google.golang.org/cloud/internal/transport.NewHTTPClient(0x7fe74318b6b8, 0xc820072d78, 0xc820547a38, 0x3, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/xx/xx/Godeps/_workspace/src/google.golang.org/cloud/internal/transport/dial.go:64 +0x2c1 fp=0xc8205478a8 sp=0xc820547780google.golang.org/cloud/pubsub.NewClient(0x7fe74318b6b8, 0xc820072d78, 0x7fff2e3248f2, 0x7, 0x0, 0x0, 0x0, 0xc800000002, 0x0, 0x0)
/go/src/xx/xx/Godeps/_workspace/src/google.golang.org/cloud/pubsub/pubsub.go:62 +0x45e fp=0xc820547a70 sp=0xc8205478a8
main.main()
/go/src/xx/xx/main.go:179 +0xc34 fp=0xc820547f10 sp=0xc820547a70
runtime.main()
/usr/local/go/src/runtime/proc.go:188 +0x2b0 fp=0xc820547f60 sp=0xc820547f10
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820547f68 sp=0xc820547f60

It works locally if I use a json key file.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#229

azr · 2016-03-22T17:02:47Z

The service is continuously built and deployed to a kubernetes cluster using wercker.

Dockerfile for build is : az3r/golang-rocksdb:1.6-4.4

Dockerfile at runtime is google/cloud-sdk + the binary

rocksdb is embeded in binary I believe ( build command is : GOOS=linux GOARCH=amd64 godep go build --ldflags '-extldflags "-static"' -a -tags netgo -o hotbase . )

Panic at startup happens a lot:

$ kubectl get pods
NAME                            READY     STATUS             RESTARTS   AGE
hotbase-newyork-b845699-tc8x2   0/1       CrashLoopBackOff   15         52min

And then suddenly starts working like a charm.

Ps: sha is 335f474

broady · 2016-03-22T19:10:36Z

That's a segfault. I think it's because there's no /etc/passwd file (and no home directory!) if you only have the binary inside the container.

If you want to use the default service account from the metadata server, you can use the ComputeTokenSource, instead of using a credential file.

broady · 2016-03-22T20:13:50Z

Very strange. I can't reproduce this.

$ for f in *; do echo ----$f----; cat $f; done
----Dockerfile----
#FROM scratch
FROM google/cloud-sdk
ADD getuser /
CMD ["/getuser"]
----getuser.go----
package main

import (
    "fmt"
    "os/user"
)

func main() {
    u, err := user.Current()
    fmt.Println(err, u)
}
----Makefile----
getuser:
    GOOS=linux GOARCH=amd64 go build --ldflags '-extldflags "-static"' -a -tags netgo -o getuser .

.PHONY: run clean

dockerbuild: getuser Dockerfile
    docker build -t broady/getuser .

run: dockerbuild
    docker run --rm broady/getuser

clean:
    rm -f getuser

broady · 2016-03-22T20:20:59Z

Are you running on Google Container Engine?

azr · 2016-03-23T10:18:02Z

Okay, I did a little manual bissection, and this is caused by the adding of rocksdb.
I used boltdb before and it just worked.

replacing boltdb with rocksdb:

local mac book pro go build
works

Now if I build the binary inside my local docker using the same commands as my builder:

If run on my local docker (google/cloud-sdk) without any credentials I get a nice :

dialing: google.DefaultTokenSource: google: could not find default credentials.

If run on my local docker (google/cloud-sdk) with GOOGLE_APPLICATION_CREDENTIALS=my-key.json

It works just fine

if run on google cloud kubernetes :

Weird thing is it can randomly work.

$ kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
hotbase-paris-6b507d4-7fupd     1/1       Running   0          35m #boltDB I just built
hotbase-paris-db746c9-5g3r0     1/1       Running   172        16h #rocksDB build from yesterday

Now the panic is always the same:

$ kubectl logs -p hotbase-paris-db746c9-5g3r0
2016-03-23 07:39:38 [hotbase] DEBUG    main
[...]
2016-03-23 07:39:44 [hotbase] DEBUG    started server
2016-03-23 07:39:44 [hotbase] DEBUG    creating pubsub client
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xe5 pc=0x7f3a2a5f8a5c]

runtime stack:
runtime.throw(0x1295d00, 0x2a)
    /usr/local/go/src/runtime/panic.go:530 +0x90
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a
runtime.execute(0x100000000, 0x0)
    /usr/local/go/src/runtime/proc.go:1791 +0x161

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0xb602b0, 0xc820513310, 0x0)
    /usr/local/go/src/runtime/cgocall.go:123 +0x11b fp=0xc8205132b8 sp=0xc820513288
os/user._Cfunc_mygetpwuid_r(0x0, 0xc8203847e0, 0x7f3a1000e4b0, 0x400, 0xc820190048, 0x0)
    ??:0 +0x44 fp=0xc820513310 sp=0xc8205132b8
os/user.lookupUnix(0x0, 0x0, 0x0, 0x1166c00, 0x0, 0x0, 0x0)
    /usr/local/go/src/os/user/lookup_unix.go:95 +0xcd8 fp=0xc8205134a8 sp=0xc820513310
os/user.current(0x4759a0, 0x0, 0x0)
    /usr/local/go/src/os/user/lookup_unix.go:38 +0x42 fp=0xc8205134e8 sp=0xc8205134a8
os/user.Current(0x54fc0d, 0x0, 0x0)
    /usr/local/go/src/os/user/lookup.go:9 +0x24 fp=0xc820513508 sp=0xc8205134e8
golang.org/x/oauth2/google.guessUnixHomeDir(0x0, 0x0)
    /go/src/xx/xx/Godeps/_workspace/src/golang.org/x/oauth2/google/sdk.go:163 +0x28 fp=0xc820513540 sp=0xc820513508
golang.org/x/oauth2/google.wellKnownFile(0x0, 0x0)
    /go/src/xx/xx/Godeps/_workspace/src/golang.org/x/oauth2/google/default.go:107 +0x36 fp=0xc8205135d8 sp=0xc820513540
golang.org/x/oauth2/google.DefaultTokenSource(0x7f3a311fa6e0, 0xc820016df8, 0xc8203ce840, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0)
    /go/src/xx/xx/Godeps/_workspace/src/golang.org/x/oauth2/google/default.go:72 +0x307 fp=0xc820513780 sp=0xc8205135d8
google.golang.org/cloud/internal/transport.NewHTTPClient(0x7f3a311fa6e0, 0xc820016df8, 0xc820513a38, 0x3, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/xx/xx/Godeps/_workspace/src/google.golang.org/cloud/internal/transport/dial.go:64 +0x2c1 fp=0xc8205138a8 sp=0xc820513780
google.golang.org/cloud/pubsub.NewClient(0x7f3a311fa6e0, 0xc820016df8, 0x7ffcdb3dc8f2, 0x7, 0x0, 0x0, 0x0, 0xc800000000, 0x0, 0x0)
    /go/src/xx/xx/Godeps/_workspace/src/google.golang.org/cloud/pubsub/pubsub.go:62 +0x45e fp=0xc820513a70 sp=0xc8205138a8
main.main()
    /go/src/xx/xx/main.go:179 +0xc34 fp=0xc820513f10 sp=0xc820513a70
runtime.main()
    /usr/local/go/src/runtime/proc.go:188 +0x2b0 fp=0xc820513f60 sp=0xc820513f10
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820513f68 sp=0xc820513f60

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 20 [chan receive]:
github.com/rcrowley/go-metrics.(*meterArbiter).tick(0x18fc8a0)
    /go/src/xx/xx/Godeps/_workspace/src/github.com/rcrowley/go-metrics/meter.go:221 +0x4b
created by github.com/rcrowley/go-metrics.NewMeter
    /go/src/xx/xx/Godeps/_workspace/src/github.com/rcrowley/go-metrics/meter.go:40 +0x154

goroutine 21 [chan receive]:
github.com/stvp/rollbar.init.1.func1()
    /go/src/xx/xx/Godeps/_workspace/src/github.com/stvp/rollbar/rollbar.go:103 +0x5c
created by github.com/stvp/rollbar.init.1
    /go/src/xx/xx/Godeps/_workspace/src/github.com/stvp/rollbar/rollbar.go:114 +0x91

goroutine 22 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 34 [select]:
net/http.(*persistConn).readLoop(0xc820101e10)
    /usr/local/go/src/net/http/transport.go:1178 +0xd52
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 59 [chan receive]:
bitbucket.org/zup/libs/librato.CaptureDebugGCStats(0x7f3a30178428, 0xc82018cf40, 0xdf8475800)
    /go/src/xx/xx/Godeps/_workspace/src/bitbucket.org/zup/libs/librato/debug.go:23 +0x5f
created by bitbucket.org/zup/libs/librato.Init
    /go/src/xx/xx/Godeps/_workspace/src/bitbucket.org/zup/libs/librato/init.go:17 +0xc2

goroutine 31 [IO wait]:
net.runtime_pollWait(0x7f3a311fbda0, 0x72, 0xc8202b0000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82024f170, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82024f170, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82024f110, 0xc8202b0000, 0x1000, 0x1000, 0x0, 0x7f3a311f6050, 0xc8200161d0)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820190258, 0xc8202b0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f3a301b4120, 0xc820190258, 0xc8201a6c98, 0xc8202b0000, 0x1000, 0x1000, 0x46d583, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8201ebb00, 0xc8202b0000, 0x1000, 0x1000, 0xc820038d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc8201a8cc0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8201a8cc0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8201a6c30)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 35 [select]:
net/http.(*persistConn).writeLoop(0xc820101e10)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 32 [select]:
net/http.(*persistConn).writeLoop(0xc8201a6c30)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 57 [IO wait]:
net.runtime_pollWait(0x7f3a311fbc20, 0x72, 0xc8202cb000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82026d870, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82026d870, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82026d810, 0xc8202cb000, 0x1000, 0x1000, 0x0, 0x7f3a311f6050, 0xc8200161d0)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82002c3b8, 0xc8202cb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc8202972c0, 0x7f3a301b4120, 0xc82002c3b8, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc8202c8000, 0x130e817, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc8202c8000, 0xc8202d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:939 +0x167
net/http.noteEOFReader.Read(0x7f3a3017fd28, 0xc8202c8000, 0xc8201a6e38, 0xc8202d3000, 0x1000, 0x1000, 0x46d583, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8203c7f00, 0xc8202d3000, 0x1000, 0x1000, 0xc82004fd0d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc82000e720)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82000e720, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8201a6dd0)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 58 [select]:
net/http.(*persistConn).writeLoop(0xc8201a6dd0)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 60 [chan receive]:
github.com/rcrowley/go-metrics.CaptureRuntimeMemStats(0x7f3a30178428, 0xc82018cf40, 0xdf8475800)
    /go/src/xx/xx/Godeps/_workspace/src/github.com/rcrowley/go-metrics/runtime.go:58 +0x5f
created by bitbucket.org/zup/libs/librato.Init
    /go/src/xx/xx/Godeps/_workspace/src/bitbucket.org/zup/libs/librato/init.go:19 +0x11d

goroutine 61 [chan receive]:
github.com/mihasya/go-metrics-librato.(*Reporter).Run(0xc820228070)
    /go/src/xx/xx/Godeps/_workspace/src/github.com/mihasya/go-metrics-librato/librato.go:45 +0x10b
github.com/mihasya/go-metrics-librato.Librato(0x7f3a30178428, 0xc82018cf40, 0x2540be400, 0x120f2e0, 0x16, 0x12b65a0, 0x40, 0xc8203b48b0, 0xf, 0xc8203b4af8, ...)
    /go/src/xx/xx/Godeps/_workspace/src/github.com/mihasya/go-metrics-librato/librato.go:39 +0xcf
created by bitbucket.org/zup/libs/librato.Init
    /go/src/xx/xx/Godeps/_workspace/src/bitbucket.org/zup/libs/librato/init.go:28 +0x206

goroutine 62 [chan receive]:
bitbucket.org/zup/libs/librato.startUptimeLoop.func1()
    /go/src/xx/xx/Godeps/_workspace/src/bitbucket.org/zup/libs/librato/init.go:38 +0x8b
created by bitbucket.org/zup/libs/librato.startUptimeLoop
    /go/src/xx/xx/Godeps/_workspace/src/bitbucket.org/zup/libs/librato/init.go:45 +0x2b

goroutine 74 [IO wait]:
net.runtime_pollWait(0x7f3a311fbb60, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82026c840, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82026c840, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82026c7e0, 0x0, 0x7f3a311fbf90, 0xc8203ce7a0)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820190018, 0x51c151, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc820190018, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2427 +0x41
net/http.(*Server).Serve(0xc8201a0300, 0x7f3a312063b0, 0xc820190018, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
net/http.(*Server).ListenAndServe(0xc8201a0300, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2098 +0x136
net/http.ListenAndServe(0x115c668, 0x5, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2195 +0x98
main.StartServer(0x115c668, 0x5)
    /go/src/xx/xx/server.go:33 +0x14a
created by main.main
    /go/src/xx/xx/main.go:174 +0xbb4

At the time of the panic no rocksdb storage is instantiated yet.

most of the time container status is :

"lastState": {
    "terminated": {
        "exitCode": 2,
        "reason": "Error",
        "startedAt": "2016-03-23T10:10:57Z",
        "finishedAt": "2016-03-23T10:11:04Z",
        "containerID": "docker://b2df12f0a7cfa90621ebf8a55fca566e1870b88c21c485618a42671c35bbe8e4"
    }
},
"ready": false,
"restartCount": 8,
"image": "gcr.io/exp_lkl/hotbase:0bc621b",
"imageID": "docker://9bc70b4b2139d4a42a61141ea6518888552ee8621f7f1088201dae78cd038fd9",
"containerID": "docker://b2df12f0a7cfa90621ebf8a55fca566e1870b88c21c485618a42671c35bbe8e4"

broady · 2016-03-23T20:12:15Z

I'll try running my demo on Container Engine.

broady · 2016-03-27T17:51:50Z

Can't reproduce this. What exactly is in your Dockerfile?

Is it running your app's binary as well as rocksdb? Is it running just rocksdb?

azr · 2016-03-29T08:23:13Z

Ah bummer, I will try to pull something reproductible !

azr · 2016-03-30T20:12:05Z

Okay, took looooong enough but I nailed it down to this code :

package main

import (
    "github.com/tecbot/gorocksdb"

    "golang.org/x/net/context"

    "google.golang.org/cloud"
    "google.golang.org/cloud/pubsub"
    "google.golang.org/cloud/storage"

    "time"
)

func main() {
    ctx := context.Background()

    gorocksdb.OpenDb(gorocksdb.NewDefaultOptions(), "./db")

    pubsub.NewClient(ctx, "my-project")

    storage.NewClient(ctx, cloud.WithScopes(storage.ScopeReadOnly))

    time.Sleep(time.Minute * 1)
    go func() {
        time.Sleep(time.Minute * 1)
    }()
    print("didn't fail !")
}

remove one of gorocksdb.OpenDb / pubsub.NewClient / storage.NewClient and it works fine.

Binary built inside a az3r/golang-rocksdb:1.6-4.4 image
Then packed into a google/cloud-sdk image ( for metadata stuff ).

This randomly panics.

broady · 2016-03-30T21:59:08Z

@azr can you show me the Dockerfiles and commands you're running, too? I don't fully understand the setup you have going on here.

azr · 2016-03-31T08:20:50Z

Okay,

For building I run:

docker run -v "$GOPATH/src/github.com/azr/test:/go/src/github.com/azr/test" -ti az3r/golang-rocksdb:1.6-4.4 bash -c "cd src/github.com/azr/test/ \\
; source storage/env-rocksdb.sh \\
&& go get . \\
&& GOOS=linux GOARCH=amd64 go build --ldflags '-extldflags "-static"' -a -tags netgo -o hotbase . "

( normally I use godep instead of go get )
Now because I did the little -v option the binary is in my cwd.

That Dockerfile I just used is in : azr/golang-rocksdb

The storage/env-rocksdb.sh file is :

#setup env variable to enable go + rocks db build
#!/bin/bash

#default snappy and leveldb install path
#you may change yourself
HOMEBREW_PREFIX=$(brew --prefix 2>/dev/null)
if [[ ! -z "$HOMEBREW_PREFIX" ]]; then
  SNAPPY_DIR=$HOMEBREW_PREFIX/opt/snappy
  LEVELDB_DIR=$HOMEBREW_PREFIX/opt/leveldb
  ROCKSDB_DIR=$HOMEBREW_PREFIX/opt/rocksdb
  LUA_DIR=$HOMEBREW_PREFIX/opt/lua51
else
  SNAPPY_DIR=/usr/local/snappy
  LEVELDB_DIR=/usr/local/leveldb
  ROCKSDB_DIR=/usr/local/rocksdb
  LUA_DIR=/usr/local/lua
fi

function add_path()
{
  # $1 path variable
  # $2 path to add
  if [ -d "$2" ] && [[ ":$1:" != *":$2:"* ]]; then
    echo "$1:$2"
  else
    echo "$1"
  fi
}

export GOPATH=$(add_path $GOPATH)

GO_BUILD_TAGS=
CGO_CFLAGS=
CGO_CXXFLAGS=
CGO_LDFLAGS=

# check dependent libray, now we only check simply, maybe later add proper checking way.

# check snappy
if [ -f $SNAPPY_DIR/include/snappy.h ]; then
    CGO_CFLAGS="$CGO_CFLAGS -I$SNAPPY_DIR/include"
    CGO_CXXFLAGS="$CGO_CXXFLAGS -I$SNAPPY_DIR/include"
    CGO_LDFLAGS="$CGO_LDFLAGS -L$SNAPPY_DIR/lib -lsnappy"
    LD_LIBRARY_PATH=$(add_path $LD_LIBRARY_PATH $SNAPPY_DIR/lib)
    DYLD_LIBRARY_PATH=$(add_path $DYLD_LIBRARY_PATH $SNAPPY_DIR/lib)
fi

# check leveldb
if [ -f $LEVELDB_DIR/include/leveldb/c.h ]; then
    CGO_CFLAGS="$CGO_CFLAGS -I$LEVELDB_DIR/include"
    CGO_CXXFLAGS="$CGO_CXXFLAGS -I$LEVELDB_DIR/include"
    CGO_LDFLAGS="$CGO_LDFLAGS -L$LEVELDB_DIR/lib -lleveldb"
    LD_LIBRARY_PATH=$(add_path $LD_LIBRARY_PATH $LEVELDB_DIR/lib)
    DYLD_LIBRARY_PATH=$(add_path $DYLD_LIBRARY_PATH $LEVELDB_DIR/lib)
    GO_BUILD_TAGS="$GO_BUILD_TAGS leveldb"
fi

# check rocksdb
if [ -f $ROCKSDB_DIR/include/rocksdb/c.h ]; then
    CGO_CFLAGS="$CGO_CFLAGS -I$ROCKSDB_DIR/include"
    CGO_CXXFLAGS="$CGO_CXXFLAGS -I$ROCKSDB_DIR/include"
    CGO_LDFLAGS="$CGO_LDFLAGS -L$ROCKSDB_DIR/lib -lrocksdb"
    LD_LIBRARY_PATH=$(add_path $LD_LIBRARY_PATH $ROCKSDB_DIR/lib)
    DYLD_LIBRARY_PATH=$(add_path $DYLD_LIBRARY_PATH $ROCKSDB_DIR/lib)
    GO_BUILD_TAGS="$GO_BUILD_TAGS rocksdb"
fi


#check lua
if [ -f $LUA_DIR/include/lua.h ]; then
    CGO_CFLAGS="$CGO_CFLAGS -I$LUA_DIR/include"
    CGO_LDFLAGS="$CGO_LDFLAGS -L$LUA_DIR/lib -llua"
    LD_LIBRARY_PATH=$(add_path $LD_LIBRARY_PATH $LUA_DIR/lib)
    DYLD_LIBRARY_PATH=$(add_path $DYLD_LIBRARY_PATH $LUA_DIR/lib)
    GO_BUILD_TAGS="$GO_BUILD_TAGS lua"
elif [ -f $LUA_DIR/include/lua-5.1/lua.h ]; then
    CGO_CFLAGS="$CGO_CFLAGS -I$LUA_DIR/include/lua-5.1"
    CGO_LDFLAGS="$CGO_LDFLAGS -L$LUA_DIR/lib -llua5.1"
    LD_LIBRARY_PATH=$(add_path $LD_LIBRARY_PATH $LUA_DIR/lib)
    DYLD_LIBRARY_PATH=$(add_path $DYLD_LIBRARY_PATH $LUA_DIR/lib)
    GO_BUILD_TAGS="$GO_BUILD_TAGS lua"
fi

export CGO_CFLAGS
export CGO_CXXFLAGS
export CGO_LDFLAGS
export LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH
export GO_BUILD_TAGS

Now that I have the binary I run :

docker build -t gcr.io/my_proj/hotbase:test . && gcloud docker -- push gcr.io/my_proj/hotbase:test && kubectl delete rc test && kubectl create -f test.yaml

That second docker file is :

FROM google/cloud-sdk

EXPOSE 8080

CMD ["/pipeline/source/hotbase"]

ADD hotbase /pipeline/source/hotbase

And the test.yaml is a replication controller:

//...
    spec:
      containers:
      - command:
        - ./pipeline/source/hotbase
        image: gcr.io/my_proj/hotbase:test
        imagePullPolicy: Always

azr · 2016-03-31T12:35:49Z

I will try pinging @tecbot also as he might have a clue.

azr · 2016-04-05T16:23:28Z

could you reproduce it @broady ?

grosskur · 2016-08-17T07:19:57Z

I think this is glibc bug 19341, which is caused by statically linking glibc. See golang/go#13470 for background.

A workaround is to link glibc dynamically, i.e., remove -ldflags '-extldflags "-static"' from the build command. See cockroachdb/cockroach#3392.

jba · 2016-09-11T12:11:52Z

@azr, did the workaround fix the problem for you?

Since this has been attributed to a bug elsewhere, and there is no contradicting evidence, I'm going to close.

azr · 2016-09-12T09:02:16Z

Hey people, thanks for your time.
I am very sorry, I changed company and we dont use gcp or k8s here so I can't tell you.

I'll forward your hotfix to them though they probably can do something with it !?

azr · 2016-09-12T09:04:23Z

Oh, actually reading from my comments, I already did that.

The whole build command was :

GOOS=linux GOARCH=amd64 go build --ldflags '-extldflags "-static"' -a -tags netgo

the netgo tag fixed another C problem

azr changed the title ~~pubsub: weird panic~~ pubsub: weird panic ( pubsub.NewClient ) Mar 22, 2016

broady self-assigned this Mar 22, 2016

dsymonds added the api: pubsub Issues related to the Pub/Sub API. label Apr 15, 2016

jba added the type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. label Jun 23, 2016

jba closed this as completed Sep 11, 2016

yoshi-automation added 🚨 This issue needs some love. triage me I really want to be triaged. labels Apr 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pubsub: weird panic ( pubsub.NewClient ) #229

pubsub: weird panic ( pubsub.NewClient ) #229

azr commented Mar 22, 2016

broady commented Mar 22, 2016

azr commented Mar 22, 2016

broady commented Mar 22, 2016

broady commented Mar 22, 2016

broady commented Mar 22, 2016

azr commented Mar 23, 2016

broady commented Mar 23, 2016

broady commented Mar 27, 2016

azr commented Mar 29, 2016

azr commented Mar 30, 2016

broady commented Mar 30, 2016

azr commented Mar 31, 2016

azr commented Mar 31, 2016

azr commented Apr 5, 2016

grosskur commented Aug 17, 2016

jba commented Sep 11, 2016

azr commented Sep 12, 2016 •

edited

Loading

azr commented Sep 12, 2016 •

edited

Loading

pubsub: weird panic ( pubsub.NewClient ) #229

pubsub: weird panic ( pubsub.NewClient ) #229

Comments

azr commented Mar 22, 2016

broady commented Mar 22, 2016

azr commented Mar 22, 2016

broady commented Mar 22, 2016

broady commented Mar 22, 2016

broady commented Mar 22, 2016

azr commented Mar 23, 2016

broady commented Mar 23, 2016

broady commented Mar 27, 2016

azr commented Mar 29, 2016

azr commented Mar 30, 2016

broady commented Mar 30, 2016

azr commented Mar 31, 2016

azr commented Mar 31, 2016

azr commented Apr 5, 2016

grosskur commented Aug 17, 2016

jba commented Sep 11, 2016

azr commented Sep 12, 2016 • edited Loading

azr commented Sep 12, 2016 • edited Loading

azr commented Sep 12, 2016 •

edited

Loading

azr commented Sep 12, 2016 •

edited

Loading