Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

recieve: Panics #1594

Closed
dtrejod opened this issue Oct 1, 2019 · 1 comment · Fixed by #1595
Closed

recieve: Panics #1594

dtrejod opened this issue Oct 1, 2019 · 1 comment · Fixed by #1595

Comments

@dtrejod
Copy link
Contributor

dtrejod commented Oct 1, 2019

Thanos, Prometheus and Golang version used

$ ./thanos --version
thanos, version 0.7.0 (branch: master, revision: baf2f3a03757c0260250d93aa8ef41a79a829bb6)
  build user:       circleci@54f32563bbc6
  build date:       20191001-13:19:04
  go version:       go1.12.5

Prometheus version 2.12.0

What happened

Go routine panic.

What you expected to happen

No go routine panic

How to reproduce it (as minimally and precisely as possible):
Simply start a receiver --

./thanos receive --log.format=logfmt --log.level=debug --tsdb.path=./test/data --objstore.config-file=./test/bucket_config.yml --receive.replication-factor=1

Full logs to relevant components

Logs

level=debug ts=2019-10-01T17:51:34.346053081Z caller=main.go:122 msg="maxprocs: Leaving GOMAXPROCS=[8]: CPU quota undefined" level=info ts=2019-10-01T17:51:34.346457856Z caller=main.go:170 msg="Tracing will be disabled" level=warn ts=2019-10-01T17:51:34.346500366Z caller=receive.go:143 component=receive msg="setting up receive; the Thanos receive component is EXPERIMENTAL, it may break significantly without notice" level=debug ts=2019-10-01T17:51:34.346746109Z caller=receive.go:192 component=receive msg="setting up tsdb" level=info ts=2019-10-01T17:51:34.350840387Z caller=head.go:509 component=receive component=tsdb component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2019-10-01T17:51:34.351074421Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 level=info ts=2019-10-01T17:51:34.35128205Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 level=info ts=2019-10-01T17:51:34.351416083Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 level=info ts=2019-10-01T17:51:34.352002711Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 level=info ts=2019-10-01T17:51:34.352230301Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 level=info ts=2019-10-01T17:51:34.360157392Z caller=head.go:509 component=receive component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2019-10-01T17:51:34.360413948Z caller=head.go:557 component=receive component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 level=info ts=2019-10-01T17:51:34.360588495Z caller=head.go:557 component=receive component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 level=info ts=2019-10-01T17:51:34.360715952Z caller=head.go:557 component=receive component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 level=info ts=2019-10-01T17:51:34.360832443Z caller=head.go:557 component=receive component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 level=info ts=2019-10-01T17:51:34.360974741Z caller=head.go:557 component=receive component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 level=info ts=2019-10-01T17:51:34.374679961Z caller=head.go:509 component=receive component=tsdb component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2019-10-01T17:51:34.375273668Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 level=info ts=2019-10-01T17:51:34.3757924Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 level=info ts=2019-10-01T17:51:34.375991581Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 level=info ts=2019-10-01T17:51:34.376171243Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 level=info ts=2019-10-01T17:51:34.37632788Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 level=info ts=2019-10-01T17:51:34.37643955Z caller=head.go:557 component=receive component=tsdb component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 level=debug ts=2019-10-01T17:51:34.37734948Z caller=receive.go:250 component=receive msg="setting up hashring" level=debug ts=2019-10-01T17:51:34.37738744Z caller=receive.go:302 component=receive msg="setting up http server" level=debug ts=2019-10-01T17:51:34.377648418Z caller=receive.go:308 component=receive msg="setting up grpc server" level=debug ts=2019-10-01T17:51:34.377683868Z caller=receive.go:353 component=receive msg="setting up receive http handler" level=info ts=2019-10-01T17:51:34.377706991Z caller=factory.go:39 component=receive msg="loading bucket configuration" level=info ts=2019-10-01T17:51:34.400789386Z caller=receive.go:432 component=receive msg="starting receiver" panic: send on closed channel

goroutine 481 [running]:
main.runReceive.func5(0xc000155440, 0x0)
/go/src/github.com/thanos-io/thanos/cmd/thanos/receive.go:269 +0x77
github.com/oklog/run.(*Group).Run.func1(0xc00298ec00, 0xc0024900f0, 0xc0000b30e0)
/go/pkg/mod/github.com/oklog/[email protected]/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
/go/pkg/mod/github.com/oklog/[email protected]/group.go:37 +0xbe

Anything else we need to know
Issue arose since merging - #1480

CC @squat
@squat
Copy link
Member

squat commented Oct 1, 2019 via email

squat added a commit to squat/thanos that referenced this issue Oct 1, 2019
@squat
cmd/thanos/receive: fix close chan panic
6abe080 
This commit fixes a panic caused by sending on the chan which is closed
too early. This chan was closed when the setup func returned, rather
than when the group's goroutine returned.

Fixes: thanos-io#1594
Signed-off-by: Lucas Servén Marín <[email protected]>
@squat squat mentioned this issue Oct 1, 2019
Merged
squat added a commit to squat/thanos that referenced this issue Oct 1, 2019
@squat
cmd/thanos/receive: fix close chan panic
e7d8707 
This commit fixes a panic caused by sending on the chan which is closed
too early. This chan was closed when the setup func returned, rather
than when the group's goroutine returned.

Fixes: thanos-io#1594
Signed-off-by: Lucas Servén Marín <[email protected]>
brancz pushed a commit that referenced this issue Oct 2, 2019
@squat @brancz
cmd/thanos/receive: fix close chan panic (#1595)
4c7ecd3 
* cmd/thanos/receive: fix close chan panic

This commit fixes a panic caused by sending on the chan which is closed
too early. This chan was closed when the setup func returned, rather
than when the group's goroutine returned.

Fixes: #1594
Signed-off-by: Lucas Servén Marín <[email protected]>

* cmd/thanos/receive: correctly clean up uploader

This commit fixes how the uploader is cleaned up. Currently, it was
cleaned up in a defer block that was run before the run groups. This
ensures the cleanup occurs only when the run groups are returning.

Signed-off-by: Lucas Servén Marín <[email protected]>
GiedriusS pushed a commit that referenced this issue Oct 28, 2019
@squat @GiedriusS
cmd/thanos/receive: fix close chan panic (#1595)
0e512fc 
* cmd/thanos/receive: fix close chan panic

This commit fixes a panic caused by sending on the chan which is closed
too early. This chan was closed when the setup func returned, rather
than when the group's goroutine returned.

Fixes: #1594
Signed-off-by: Lucas Servén Marín <[email protected]>

* cmd/thanos/receive: correctly clean up uploader

This commit fixes how the uploader is cleaned up. Currently, it was
cleaned up in a defer block that was run before the run groups. This
ensures the cleanup occurs only when the run groups are returning.

Signed-off-by: Lucas Servén Marín <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cmd/thanos/receive: fix close chan panic squat/thanos
2 participants
@dtrejod @squat