Collector with badger storage failed to start after crash

[ont]

Requirement

Stable recovering of jaeger docker container all-in-one with badger storage backend after crash or OOM kill.

Problem

Jaeger docker container (all-in-one) failed to start. Following error occurs:

{"level":"fatal","ts":1570224769.31795,"caller":"all-in-one/main.go:105","msg":"Failed to init storage factory","error":"Unable to replay value log: \"/badger/data/000000.vlog\": Value log truncate required to run DB. This might result in data loss.","errorVerbose":"Value log truncate required to run DB. This might result in data loss.\ngithub.com/jaegertracing/jaeger/vendor/github.com/dgraph-io/badger.init.ializers\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/dgraph-io/badger/errors.go:98\nruntime.main\n\t/home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/proc.go:188\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/asm_amd64.s:1337\nUnable to replay value log: \"/badger/data/000000.vlog\"\ngithub.com/jaegertracing/jaeger/vendor/github.com/dgraph-io/badger.(*valueLog).Replay\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/dgraph-io/badger/value.go:772\ngithub.com/jaegertracing/jaeger/vendor/github.com/dgraph-io/badger.Open\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/dgraph-io/badger/db.go:306\ngithub.com/jaegertracing/jaeger/plugin/storage/badger.(*Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/badger/factory.go:115\ngithub.com/jaegertracing/jaeger/plugin/storage.(*Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/factory.go:108\nmain.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/all-in-one/main.go:104\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/all-in-one/main.go:171\nruntime.main\n\t/home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/proc.go:200\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/asm_amd64.s:1337","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/all-in-one/main.go:105\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/all-in-one/main.go:171\nruntime.main\n\t/home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/proc.go:200"}

Proposal

Provide option for enabling truncate in badger. See: dgraph-io/badger#744

[objectiser]

cc @burmanm