v0.5.0 panics with invalid memory address or nil pointer dereference

[dneuhaeuser-zalando]

Error message from the log:

2018/05/28 14:45:15 http: panic serving [::1]:38530: runtime error: invalid memory address or nil pointer dereference
goroutine 1797 [running]:
net/http.(*conn).serve.func1(0xc42041e140)
	/usr/local/go/src/net/http/server.go:1726 +0xd0
panic(0x888920, 0xc07ea0)
	/usr/local/go/src/runtime/panic.go:502 +0x229
github.com/prometheus/pushgateway/storage.(*DiskMetricStore).GetMetricFamilies(0xc42001ca50, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/pushgateway/storage/diskmetricstore.go:113 +0x4af
main.main.func1(0x1, 0x0, 0x8d39e0, 0xc42032e1b8, 0xc420078010)
	/go/src/github.com/prometheus/pushgateway/main.go:80 +0x2a
github.com/prometheus/pushgateway/vendor/github.com/prometheus/client_golang/prometheus.GathererFunc.Gather(0xc42022c330, 0x0, 0x0, 0x0, 0x1, 0xc42032e1b8)
	/go/src/github.com/prometheus/pushgateway/vendor/github.com/prometheus/client_golang/prometheus/registry.go:171 +0x27
github.com/prometheus/pushgateway/vendor/github.com/prometheus/client_golang/prometheus.Gatherers.Gather(0xc4200ebea0, 0x2, 0x2, 0x8ca0c0, 0x410079, 0x10000c42046c090, 0x73000000000008, 0x8)
	/go/src/github.com/prometheus/pushgateway/vendor/github.com/prometheus/client_golang/prometheus/registry.go:591 +0xe2e
github.com/prometheus/pushgateway/vendor/github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x97c0a0, 0xc42032e048, 0xc42019c200)
	/go/src/github.com/prometheus/pushgateway/vendor/github.com/prometheus/client_golang/prometheus/http.go:75 +0x47
net/http.HandlerFunc.ServeHTTP(0x93cc88, 0x97c0a0, 0xc42032e048, 0xc42019c200)
	/usr/local/go/src/net/http/server.go:1947 +0x44
net/http.(Handler).ServeHTTP-fm(0x97c0a0, 0xc42032e048, 0xc42019c200)
	/usr/local/go/src/net/http/h2_bundle.go:5475 +0x4d
github.com/prometheus/pushgateway/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x97c660, 0xc42049c1c0, 0xc42019c200)
	/go/src/github.com/prometheus/pushgateway/vendor/github.com/prometheus/client_golang/prometheus/http.go:296 +0x229
net/http.HandlerFunc.ServeHTTP(0xc42001d4a0, 0x97c660, 0xc42049c1c0, 0xc42019c200)
	/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/prometheus/pushgateway/vendor/github.com/julienschmidt/httprouter.(*Router).Handler.func1(0x97c660, 0xc42049c1c0, 0xc42019c100, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/pushgateway/vendor/github.com/julienschmidt/httprouter/params_go17.go:26 +0x137
github.com/prometheus/pushgateway/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc42022a940, 0x97c660, 0xc42049c1c0, 0xc42019c100)
	/go/src/github.com/prometheus/pushgateway/vendor/github.com/julienschmidt/httprouter/router.go:334 +0x79c
net/http.serverHandler.ServeHTTP(0xc4201aa000, 0x97c660, 0xc42049c1c0, 0xc42019c100)
	/usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc42041e140, 0x97cd60, 0xc42022a440)
	/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2795 +0x27b

This seems to reliably occur with every request to /metrics since upgrading to 0.5.0 from 0.4.0.

[beorn7]

Urgh, I see what's going wrong. It has to do with the legacy format conversion.

To fix properly, I'll first write a test for the conversion.

To get your PGW running, you need to start from an empty state. If I understand the problem correctly, the failed legacy conversion has thrown away your metric data now anyway. (With the upcoming fixed version, you could start it from a pre-0.5 backup.)

[dneuhaeuser-zalando]

We're ok with losing data here (on rare occasions such as this one). We've already rolled back to 0.4.0, expecting to lose the metrics or to have to delete the file but the rollback seems to have worked just fine with no data loss.

[beorn7]

Right, that makes sense (now that I think about it). The new version will still gob the unconverted legacy metric to the storage. Cool. So rollback is the way to go.