Store: initialisation time

[xjewer]

During the store initialisation process neither grpc or http server aren't running. Hence there isn't any approach to use liveness k8s probe to check if container in pod is running.

thanos, version 0.1.0-rc.2 (branch: master, revision: a4128358e9df68170a4df7f0d4c7bef2bab2c3ca)
  build user:       circleci@36ffc8669a55
  build date:       20180718-06:16:19
  go version:       go1.10.3

What you expected to happen

Easy way is to run http server with a metrics endpoint (and debug) before an initialisation process. So k8s can use rollingUpdate properly with liveness and readiness probes

https://github.com/improbable-eng/thanos/blob/master/cmd/thanos/store.go#L178

Full logs to relevant components

Logs

level=info ts=2018-09-21T22:07:44.962137265Z caller=flags.go:64 msg="StoreAPI address that will be propagated through gossip" address=10.36.179.93:10901
level=debug ts=2018-09-21T22:07:44.97119429Z caller=cluster.go:142 component=cluster msg="resolved peers to following addresses" peers=
level=debug ts=2018-09-21T22:07:44.972085125Z caller=store.go:112 msg="initializing bucket store"
.....
level=debug ts=2018-09-21T22:11:53.760590579Z caller=store.go:116 msg="bucket store ready" init_duration=4m8.788504567s
level=info ts=2018-09-21T22:11:53.760823559Z caller=store.go:176 msg="starting store node"
level=info ts=2018-09-21T22:11:53.760869156Z caller=store.go:145 msg="Listening for StoreAPI gRPC" address=0.0.0.0:10901
level=info ts=2018-09-21T22:11:53.760904867Z caller=main.go:243 msg="Listening for metrics" address=0.0.0.0:10902

[bwplotka]

This was fixed (: