Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WAL checkpoint causes panics #6512

Closed
simonswine opened this issue Jun 27, 2022 · 0 comments · Fixed by #6516
Closed

WAL checkpoint causes panics #6512

simonswine opened this issue Jun 27, 2022 · 0 comments · Fixed by #6516
Labels
component/ingester type/bug Somehing is not working as expected

Comments

@simonswine
Copy link
Contributor

simonswine commented Jun 27, 2022

Describe the bug

During WAL checkpointing I have encountered a panic

Expected behavior

  • Do not panic

Environment:

  • Infrastructure: At Grafana Labs
  • Version: weekly-k102-6b9361d0

Screenshots, Promtail config, or terminal output*

level=info ts=2022-06-27T14:36:10.692694002Z caller=checkpoint.go:615 msg="starting checkpoint"
level=info ts=2022-06-27T14:36:10.693230616Z caller=checkpoint.go:340 msg="attempting checkpoint for" dir=/loki/wal/checkpoint.006820
level=info ts=2022-06-27T14:36:10.965299281Z caller=checkpoint.go:573 msg="checkpoint done" time=263.476585ms
panic: runtime error: index out of range [1540] with length 1540

goroutine 1178 [running]:
github.com/grafana/loki/pkg/ingester.newStreamsIterator({0x295c800, 0xc000e06580})
        /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/checkpoint.go:216 +0x24e
github.com/grafana/loki/pkg/ingester.(*ingesterSeriesIter).Iter(0x1f2a73a)
        /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/checkpoint.go:189 +0x25
github.com/grafana/loki/pkg/ingester.(*Checkpointer).PerformCheckpoint(0xc0109f9f78)
        /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/checkpoint.go:577 +0x25f
github.com/grafana/loki/pkg/ingester.(*Checkpointer).Run(0xc0109f9f78)
        /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/checkpoint.go:616 +0x21c
github.com/grafana/loki/pkg/ingester.(*walWrapper).run(0xc000b44690)
        /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/wal.go:166 +0x21c
created by github.com/grafana/loki/pkg/ingester.(*walWrapper).Start
        /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/wal.go:102 +0x6f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/ingester type/bug Somehing is not working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants