Handle SIGHUP event

[tomleb]

Thanos, Prometheus and Golang version used

thanos: v0.5.0

What happened

Sending a SIGHUP signal to sidecar, store gateway, querier or compactor crashes it.

What you expected to happen

They should not crash. Maybe they should also reload configuration (?)

How to reproduce it (as minimally and precisely as possible):

$ kill -SIGHUP <thanos>

I think thanos should at least not crash upon receiving a SIGHUP signal.

I understand that flags are used to configure thanos components, but it would be nice to have a way to reload configuration files immediately with a SIGHUP signal. This is what most applications do (eg: prometheus). Potential use case: Have an ansible handler sending the SIGHUP signal.

[daixiang0]

It is fixed in 0.9:

level=info ts=2019-12-23T03:16:32.298801119Z caller=grpc.go:98 service=gRPC/server component=store msg="listening for StoreAPI gRPC" address=0.0.0.0:10901
level=info ts=2019-12-23T03:16:32.298299889Z caller=prober.go:127 msg="changing probe status" status=healthy
level=info ts=2019-12-23T03:16:32.302900836Z caller=http.go:53 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
Hangup
[root@test thanos]# 
───────────────────────────────────────────────────────────────────────────────────────────[root@test thanos]# ps faux | grep than
root       61482  0.7  0.1 724020 30028 pts/2    Sl+  11:16   0:00  |   \_ ./thanos store --objstore.config={type: FILESYSTEM, config: {directory: ./obj_data}}
root       61929  0.0  0.0 112708   992 pts/3    S+   11:16   0:00      \_ grep --color=auto than
[root@test thanos]# kill -SIGHUP 61482
[root@test thanos]# 

@jojohappy @bwplotka please close it.

[jojohappy]

@daixiang0 Thanks! let me double check it ASAP.

[jojohappy]

@daixiang0 Hi, maybe this issue is not fixed. @tomleb hopes that thanos should not exit when receive the SIGHUB.

@tomleb As you said, thanos now use the flags as the provider for the most configurations, if you want to change them, restart it. On the other hand, it is a nice improvement to reload the configuration file of object store without restart. Maybe we can use the reload package to listen the changes of the configuration file of object store like store discovery files.

[stale]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[bwplotka]

Yea we might not handle it properly. I think it's a feature to reloader or just ignore SIGHUP. Help wanted (: