From 059759cc5767a67a631f8dc3ad00e9052fd4fb3f Mon Sep 17 00:00:00 2001 From: Martin Chodur Date: Sat, 23 Mar 2019 00:10:19 +0100 Subject: [PATCH] feat: added /-/healthy and /-/ready endppoints Signed-off-by: Martin Chodur --- CHANGELOG.md | 13 +- cmd/thanos/bucket.go | 4 +- cmd/thanos/compact.go | 26 +-- cmd/thanos/downsample.go | 8 +- cmd/thanos/main.go | 12 +- cmd/thanos/main_test.go | 63 +++++++ cmd/thanos/query.go | 22 ++- cmd/thanos/receive.go | 89 +++++---- cmd/thanos/rule.go | 19 +- cmd/thanos/sidecar.go | 30 ++- cmd/thanos/store.go | 25 ++- go.sum | 38 +--- pkg/component/component.go | 5 + pkg/prober/prober.go | 157 ++++++++++++++++ pkg/prober/prober_test.go | 172 ++++++++++++++++++ pkg/receive/handler.go | 125 ++++--------- scripts/quickstart.sh | 3 +- test/e2e/spinup_test.go | 8 +- .../manifests/prometheus-ha-sidecar-lts.yaml | 8 + .../manifests/prometheus-ha-sidecar.yaml | 8 + .../manifests/thanos-compactor.yaml | 8 + .../manifests/thanos-querier-no-us1.yaml | 4 + .../manifests/thanos-querier.yaml | 4 + .../manifests/thanos-ruler.yaml | 10 +- .../manifests/thanos-store-gateway.yaml | 8 + 25 files changed, 643 insertions(+), 226 deletions(-) create mode 100644 cmd/thanos/main_test.go create mode 100644 pkg/prober/prober.go create mode 100644 pkg/prober/prober_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 41d6a97a50b..8fe2bbca103 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,8 @@ We use *breaking* word for marking changes that are not backward compatible (rel ### Added - [#811](https://github.com/improbable-eng/thanos/pull/811) Remote write receiver +- [#644](https://github.com/improbable-eng/thanos/issues/644) Added `/-/healthy` and `/-/ready` endpoints to all node types +### Fixed - [#798](https://github.com/improbable-eng/thanos/pull/798) Ability to limit the maximum concurrent about of Series() calls in Thanos Store and the maximum amount of samples. - [#910](https://github.com/improbable-eng/thanos/pull/910) Query's stores UI page is now sorted by type and old DNS or File SD stores are removed after 5 minutes (configurable via the new `--store.unhealthy-timeout=5m` flag). @@ -20,7 +22,7 @@ New options: * `--store.grpc.series-sample-limit` limits the amount of samples that might be retrieved on a single Series() call. By default it is 0. Consider enabling it by setting it to more than 0 if you are running on limited resources. * `--store.grpc.series-max-concurrency` limits the number of concurrent Series() calls in Thanos Store. By default it is 20. Considering making it lower or bigger depending on the scale of your deployment. -* `--index.generate-missing-cache-file` if enabled, on startup compactor runs an on-off job that scans all the blocks to find all blocks with missing index cache file. It generates those if needed and upload. By default is disabled. Check logs on existence the line `generating index cache files is done`, then you can disable this flag. +* `--index.generate-missing-cache-file` if enabled, on startup compactor runs an on-off job that scans all the blocks to find all blocks with missing index cache file. It generates those if needed and upload. By default is disabled. Check logs on existence the line `generating index cache files is done`, then you can disable this flag. New metrics: * `thanos_bucket_store_queries_dropped_total` shows how many queries were dropped due to the samples limit; @@ -34,17 +36,18 @@ New tracing span: :warning: **WARNING** :warning: #798 adds a new default limit to Thanos Store: `--store.grpc.series-max-concurrency`. Most likely you will want to make it the same as `--query.max-concurrent` on Thanos Query. - [#970](https://github.com/improbable-eng/thanos/pull/970) Added `PartialResponseStrategy` field for `RuleGroups` for `Ruler`. -- [#1016](https://github.com/improbable-eng/thanos/pull/1016) Added option for another DNS resolver (miekg/dns client). +- [#1016](https://github.com/improbable-eng/thanos/pull/1016) Added option for another DNS resolver (miekg/dns client). This to have SRV resolution working on [Golang 1.11+ with KubeDNS below v1.14](https://github.com/golang/go/issues/27546) -- [#986](https://github.com/improbable-eng/thanos/pull/986) Store index cache files in object storage, reduces store start-up time by skipping the generating the index cache for all blocks and only do this for recently created uncompacted blocks. +- [#986](https://github.com/improbable-eng/thanos/pull/986) Store index cache files in object storage, reduces store start-up time by skipping the generating the index cache for all blocks and only do this for recently created uncompacted blocks. -### Changed +### Changed - [#970](https://github.com/improbable-eng/thanos/pull/970) Deprecated partial_response_disabled proto field. Added partial_response_strategy instead. Both in gRPC and Query API. - [#970](https://github.com/improbable-eng/thanos/pull/970) No `PartialResponseStrategy` field for `RuleGroups` by default means `abort` strategy (old PartialResponse disabled) as this is recommended option for Rules and alerts. ### Fixed - [#921](https://github.com/improbable-eng/thanos/pull/921) `thanos_objstore_bucket_last_successful_upload_time` now does not appear when no blocks have been uploaded so far -- [#966](https://github.com/improbable-eng/thanos/pull/966) Bucket: verify no longer warns about overlapping blocks, that overlap `0s` +- [#966](https://github.com/improbable-eng/thanos/pull/966) Bucket: verify no longer warns about overlapping blocks, that overlap `0s` +- [#921](https://github.com/improbable-eng/thanos/pull/921) `thanos_objstore_bucket_last_successful_upload_time` now does not appear when no blocks have been uploaded so far ## [v0.3.2](https://github.com/improbable-eng/thanos/releases/tag/v0.3.2) - 2019.03.04 diff --git a/cmd/thanos/bucket.go b/cmd/thanos/bucket.go index e7a1c220408..d81643d21a8 100644 --- a/cmd/thanos/bucket.go +++ b/cmd/thanos/bucket.go @@ -20,13 +20,13 @@ import ( "github.com/oklog/run" "github.com/oklog/ulid" "github.com/olekukonko/tablewriter" - "github.com/opentracing/opentracing-go" + opentracing "github.com/opentracing/opentracing-go" "github.com/pkg/errors" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/tsdb/labels" "golang.org/x/text/language" "golang.org/x/text/message" - "gopkg.in/alecthomas/kingpin.v2" + kingpin "gopkg.in/alecthomas/kingpin.v2" ) var ( diff --git a/cmd/thanos/compact.go b/cmd/thanos/compact.go index 89cf8cb07a1..c299d9f0417 100644 --- a/cmd/thanos/compact.go +++ b/cmd/thanos/compact.go @@ -17,15 +17,17 @@ import ( "github.com/improbable-eng/thanos/pkg/block/metadata" "github.com/improbable-eng/thanos/pkg/compact" "github.com/improbable-eng/thanos/pkg/compact/downsample" + "github.com/improbable-eng/thanos/pkg/component" "github.com/improbable-eng/thanos/pkg/objstore" "github.com/improbable-eng/thanos/pkg/objstore/client" + "github.com/improbable-eng/thanos/pkg/prober" "github.com/improbable-eng/thanos/pkg/runutil" "github.com/oklog/run" - "github.com/opentracing/opentracing-go" + opentracing "github.com/opentracing/opentracing-go" "github.com/pkg/errors" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/tsdb" - "gopkg.in/alecthomas/kingpin.v2" + kingpin "gopkg.in/alecthomas/kingpin.v2" ) var ( @@ -124,7 +126,6 @@ func registerCompact(m map[string]setupFunc, app *kingpin.Application, name stri compact.ResolutionLevel5m: time.Duration(*retention5m), compact.ResolutionLevel1h: time.Duration(*retention1h), }, - name, *disableDownsampling, *maxCompactionLevel, *blockSyncConcurrency, @@ -146,7 +147,6 @@ func runCompact( wait bool, generateMissingIndexCacheFiles bool, retentionByResolution map[compact.ResolutionLevel]time.Duration, - component string, disableDownsampling bool, maxCompactionLevel int, blockSyncConcurrency int, @@ -167,12 +167,12 @@ func runCompact( confContentYaml, err := objStoreConfig.Content() if err != nil { - return err + return errors.Wrap(err, "loading flag content") } - bkt, err := client.NewBucket(logger, confContentYaml, reg, component) + bkt, err := client.NewBucket(logger, confContentYaml, reg, component.Compact.String()) if err != nil { - return err + return errors.Wrap(err, "initializing bucket") } // Ensure we close up everything properly. @@ -204,6 +204,12 @@ func runCompact( return errors.Wrap(err, "create compactor") } + readinessProber := prober.NewProber(component.Compact, logger) + err = metricHTTPListenGroup(g, logger, reg, httpBindAddr, *readinessProber) + if err != nil { + return errors.Wrap(err, "create readiness prober") + } + var ( compactDir = path.Join(dataDir, "compact") downsamplingDir = path.Join(dataDir, "downsample") @@ -307,14 +313,10 @@ func runCompact( return errors.Wrap(err, "error executing compaction") }) - }, func(error) { + }, func(err error) { cancel() }) - if err := metricHTTPListenGroup(g, logger, reg, httpBindAddr); err != nil { - return err - } - level.Info(logger).Log("msg", "starting compact node") return nil } diff --git a/cmd/thanos/downsample.go b/cmd/thanos/downsample.go index 728dbdbf4a7..06af9b20022 100644 --- a/cmd/thanos/downsample.go +++ b/cmd/thanos/downsample.go @@ -19,12 +19,12 @@ import ( "github.com/improbable-eng/thanos/pkg/runutil" "github.com/oklog/run" "github.com/oklog/ulid" - "github.com/opentracing/opentracing-go" + opentracing "github.com/opentracing/opentracing-go" "github.com/pkg/errors" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/tsdb" "github.com/prometheus/tsdb/chunkenc" - "gopkg.in/alecthomas/kingpin.v2" + kingpin "gopkg.in/alecthomas/kingpin.v2" ) func registerDownsample(m map[string]setupFunc, app *kingpin.Application, name string) { @@ -49,12 +49,12 @@ func runDownsample( ) error { confContentYaml, err := objStoreConfig.Content() if err != nil { - return err + return errors.Wrap(err, "loading flag content") } bkt, err := client.NewBucket(logger, confContentYaml, reg, component.Downsample.String()) if err != nil { - return err + return errors.Wrap(err, "initializing bucket") } // Ensure we close up everything properly. diff --git a/cmd/thanos/main.go b/cmd/thanos/main.go index 2dbbad88a33..ddddbd77548 100644 --- a/cmd/thanos/main.go +++ b/cmd/thanos/main.go @@ -18,17 +18,17 @@ import ( "syscall" gmetrics "github.com/armon/go-metrics" - gprom "github.com/armon/go-metrics/prometheus" "github.com/go-kit/kit/log" "github.com/go-kit/kit/log/level" grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware" grpc_recovery "github.com/grpc-ecosystem/go-grpc-middleware/recovery" grpc_prometheus "github.com/grpc-ecosystem/go-grpc-prometheus" + "github.com/improbable-eng/thanos/pkg/prober" "github.com/improbable-eng/thanos/pkg/runutil" "github.com/improbable-eng/thanos/pkg/tracing" "github.com/oklog/run" - "github.com/opentracing/opentracing-go" + opentracing "github.com/opentracing/opentracing-go" "github.com/pkg/errors" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp" @@ -295,10 +295,11 @@ func defaultGRPCServerOpts(logger log.Logger, reg *prometheus.Registry, tracer o } // metricHTTPListenGroup is a run.Group that servers HTTP endpoint with only Prometheus metrics. -func metricHTTPListenGroup(g *run.Group, logger log.Logger, reg *prometheus.Registry, httpBindAddr string) error { +func metricHTTPListenGroup(g *run.Group, logger log.Logger, reg *prometheus.Registry, httpBindAddr string, readinessProber prober.Prober) error { mux := http.NewServeMux() registerMetrics(mux, reg) registerProfile(mux) + readinessProber.RegisterInMux(mux) l, err := net.Listen("tcp", httpBindAddr) if err != nil { @@ -307,8 +308,11 @@ func metricHTTPListenGroup(g *run.Group, logger log.Logger, reg *prometheus.Regi g.Add(func() error { level.Info(logger).Log("msg", "Listening for metrics", "address", httpBindAddr) + readinessProber.SetHealthy() + readinessProber.SetReady() return errors.Wrap(http.Serve(l, mux), "serve metrics") - }, func(error) { + }, func(err error) { + readinessProber.SetNotHealthy(err) runutil.CloseWithLogOnErr(logger, l, "metric listener") }) return nil diff --git a/cmd/thanos/main_test.go b/cmd/thanos/main_test.go new file mode 100644 index 00000000000..ee670c79549 --- /dev/null +++ b/cmd/thanos/main_test.go @@ -0,0 +1,63 @@ +package main + +import ( + "context" + "fmt" + "net/http" + "path" + "testing" + "time" + + "github.com/go-kit/kit/log" + "github.com/improbable-eng/thanos/pkg/prober" + "github.com/improbable-eng/thanos/pkg/runutil" + "github.com/improbable-eng/thanos/pkg/testutil" + "github.com/oklog/run" + "github.com/prometheus/client_golang/prometheus" +) + +type TestComponent struct { + name string +} + +func (c TestComponent) String() string { + return c.name +} + +func queryHTTPGetEndpoint(ctx context.Context, t *testing.T, logger log.Logger, url string) (*http.Response, error) { + req, err := http.NewRequest("GET", fmt.Sprintf("http://%s", url), nil) + testutil.Ok(t, err) + return http.DefaultClient.Do(req.WithContext(ctx)) +} + +func TestGenericHttpEndpoints(t *testing.T) { + var g run.Group + logger := log.NewNopLogger() + metricsRegistry := prometheus.NewRegistry() + component := TestComponent{name: "sidecar"} + ctx := context.Background() + + freePort, err := testutil.FreePort() + testutil.Ok(t, err) + + serverAddress := fmt.Sprintf("127.0.0.1:%d", freePort) + + p := prober.NewProber(component, logger) + err = metricHTTPListenGroup(&g, logger, metricsRegistry, serverAddress, *p) + testutil.Ok(t, err) + go func() { _ = g.Run() }() + + testutil.Ok(t, runutil.Retry(time.Second, ctx.Done(), func() error { + resp, err := queryHTTPGetEndpoint(ctx, t, log.NewNopLogger(), path.Join(serverAddress, "/metrics")) + testutil.Ok(t, err) + testutil.Equals(t, 200, resp.StatusCode) + return err + })) + + testutil.Ok(t, runutil.Retry(time.Second, ctx.Done(), func() error { + resp, err := queryHTTPGetEndpoint(ctx, t, log.NewNopLogger(), path.Join(serverAddress, "/debug/pprof/")) + testutil.Ok(t, err) + testutil.Equals(t, 200, resp.StatusCode) + return err + })) +} diff --git a/cmd/thanos/query.go b/cmd/thanos/query.go index 1e229bc789c..09021faf369 100644 --- a/cmd/thanos/query.go +++ b/cmd/thanos/query.go @@ -21,6 +21,7 @@ import ( "github.com/improbable-eng/thanos/pkg/discovery/cache" "github.com/improbable-eng/thanos/pkg/discovery/dns" "github.com/improbable-eng/thanos/pkg/extprom" + "github.com/improbable-eng/thanos/pkg/prober" "github.com/improbable-eng/thanos/pkg/query" v1 "github.com/improbable-eng/thanos/pkg/query/api" "github.com/improbable-eng/thanos/pkg/runutil" @@ -300,6 +301,8 @@ func runQuery( dns.ResolverType(dnsSDResolver), ) + readinessProber := prober.NewProber(component.Query, logger) + var ( stores = query.NewStoreSet( logger, @@ -340,6 +343,7 @@ func runQuery( }, ) ) + // Periodically update the store set with the addresses we see in our cluster. { ctx, cancel := context.WithCancel(context.Background()) @@ -388,6 +392,7 @@ func runQuery( close(fileSDUpdates) }) } + { ctx, cancel := context.WithCancel(context.Background()) g.Add(func() error { @@ -415,6 +420,7 @@ func runQuery( cancel() }) } + // Start query API + UI HTTP server. { router := route.New() @@ -438,12 +444,7 @@ func runQuery( api.Register(router.WithPrefix(path.Join(webRoutePrefix, "/api/v1")), tracer, logger) - router.Get("/-/healthy", func(w http.ResponseWriter, r *http.Request) { - w.WriteHeader(http.StatusOK) - if _, err := fmt.Fprintf(w, "Thanos Querier is Healthy.\n"); err != nil { - level.Error(logger).Log("msg", "Could not write health check response.") - } - }) + readinessProber.RegisterInRouter(router) mux := http.NewServeMux() registerMetrics(mux, reg) @@ -457,11 +458,14 @@ func runQuery( g.Add(func() error { level.Info(logger).Log("msg", "Listening for query and metrics", "address", httpBindAddr) + readinessProber.SetHealthy() return errors.Wrap(http.Serve(l, mux), "serve query") - }, func(error) { + }, func(err error) { + readinessProber.SetNotHealthy(err) runutil.CloseWithLogOnErr(logger, l, "query and metric listener") }) } + // Start query (proxy) gRPC StoreAPI. { l, err := net.Listen("tcp", grpcBindAddr) @@ -480,8 +484,10 @@ func runQuery( g.Add(func() error { level.Info(logger).Log("msg", "Listening for StoreAPI gRPC", "address", grpcBindAddr) + readinessProber.SetReady() return errors.Wrap(s.Serve(l), "serve gRPC") - }, func(error) { + }, func(err error) { + readinessProber.SetNotReady(err) s.Stop() runutil.CloseWithLogOnErr(logger, l, "store gRPC listener") }) diff --git a/cmd/thanos/receive.go b/cmd/thanos/receive.go index 63a3ba9c360..028b872431c 100644 --- a/cmd/thanos/receive.go +++ b/cmd/thanos/receive.go @@ -1,20 +1,24 @@ package main import ( - "context" "fmt" + stdlog "log" "net" + "net/http" "sync" "time" "github.com/go-kit/kit/log" "github.com/go-kit/kit/log/level" "github.com/improbable-eng/thanos/pkg/component" + "github.com/improbable-eng/thanos/pkg/prober" "github.com/improbable-eng/thanos/pkg/receive" "github.com/improbable-eng/thanos/pkg/runutil" "github.com/improbable-eng/thanos/pkg/store" "github.com/improbable-eng/thanos/pkg/store/storepb" + conntrack "github.com/mwitkow/go-conntrack" "github.com/oklog/run" + "github.com/opentracing-contrib/go-stdlib/nethttp" opentracing "github.com/opentracing/opentracing-go" "github.com/pkg/errors" "github.com/prometheus/client_golang/prometheus" @@ -28,10 +32,7 @@ func registerReceive(m map[string]setupFunc, app *kingpin.Application, name stri cmd := app.Command(name, "Accept Prometheus remote write API requests and write to local tsdb (EXPERIMENTAL, this may change drastically without notice)") grpcBindAddr, cert, key, clientCA := regGRPCFlags(cmd) - httpMetricsBindAddr := regHTTPAddrFlag(cmd) - - remoteWriteAddress := cmd.Flag("remote-write.address", "Address to listen on for remote write requests."). - Default("0.0.0.0:19291").String() + httpBindAddr := regHTTPAddrFlag(cmd) dataDir := cmd.Flag("tsdb.path", "Data directory of TSDB."). Default("./data").String() @@ -46,8 +47,7 @@ func registerReceive(m map[string]setupFunc, app *kingpin.Application, name stri *cert, *key, *clientCA, - *httpMetricsBindAddr, - *remoteWriteAddress, + *httpBindAddr, *dataDir, ) } @@ -62,11 +62,10 @@ func runReceive( cert string, key string, clientCA string, - httpMetricsBindAddr string, - remoteWriteAddress string, + httpBindAddr string, dataDir string, ) error { - logger = log.With(logger, "component", "receive") + logger = log.With(logger, "component", component.Receive.String()) level.Warn(logger).Log("msg", "setting up receive; the Thanos receive component is EXPERIMENTAL, it may break significantly without notice") tsdbCfg := &tsdb.Options{ @@ -78,17 +77,19 @@ func runReceive( localStorage := &tsdb.ReadyStorage{} receiver := receive.NewWriter(log.With(logger, "component", "receive-writer"), localStorage) - webHandler := receive.NewHandler(log.With(logger, "component", "receive-handler"), &receive.Options{ - Receiver: receiver, - ListenAddress: remoteWriteAddress, - Registry: reg, - ReadyStorage: localStorage, - }) // Start all components while we wait for TSDB to open but only load // initial config and mark ourselves as ready after it completed. dbOpen := make(chan struct{}) + readinessProber := prober.NewProber(component.Receive, logger) + webHandler := receive.NewHandler(log.With(logger, "component", "receive-handler"), &receive.Options{ + Receiver: receiver, + ReadinessProber: readinessProber, + Registry: reg, + ReadyStorage: localStorage, + }) + // sync.Once is used to make sure we can close the channel at different execution stages(SIGTERM or when the config is loaded). type closeOnce struct { C chan struct{} @@ -121,8 +122,6 @@ func runReceive( reloadReady.Close() - webHandler.Ready() - level.Info(logger).Log("msg", "server is ready to receive web requests.") <-cancel return nil }, @@ -165,11 +164,6 @@ func runReceive( ) } - level.Debug(logger).Log("msg", "setting up metric http listen-group") - if err := metricHTTPListenGroup(g, logger, reg, httpMetricsBindAddr); err != nil { - return err - } - level.Debug(logger).Log("msg", "setting up grpc server") { var ( @@ -199,6 +193,8 @@ func runReceive( storepb.RegisterStoreServer(s, tsdbStore) level.Info(logger).Log("msg", "listening for StoreAPI gRPC", "address", grpcBindAddr) + webHandler.Ready() + level.Info(logger).Log("msg", "server is ready to receive web requests.") return errors.Wrap(s.Serve(l), "serve gRPC") }, func(error) { if s != nil { @@ -210,20 +206,41 @@ func runReceive( }) } - level.Debug(logger).Log("msg", "setting up receive http handler") + level.Debug(logger).Log("msg", "setting up http handler") { - ctx, cancel := context.WithCancel(context.Background()) - g.Add( - func() error { - if err := webHandler.Run(ctx); err != nil { - return fmt.Errorf("error starting web server: %s", err) - } - return nil - }, - func(err error) { - cancel() - }, - ) + + mux := http.NewServeMux() + registerMetrics(mux, reg) + registerProfile(mux) + readinessProber.RegisterInMux(mux) + webHandler.HandleInMux("/", mux) + + listener, err := net.Listen("tcp", httpBindAddr) + if err != nil { + return errors.Wrapf(err, "listen HTTP on address %s", httpBindAddr) + } + + // Monitor incoming connections with conntrack. + listener = conntrack.NewListener(listener, + conntrack.TrackWithName("http"), + conntrack.TrackWithTracing()) + + operationName := nethttp.OperationNameFunc(func(r *http.Request) string { + return fmt.Sprintf("%s %s", r.Method, r.URL.Path) + }) + + httpSrv := &http.Server{ + Handler: nethttp.Middleware(opentracing.GlobalTracer(), mux, operationName), + ErrorLog: stdlog.New(log.NewStdlibAdapter(level.Error(logger)), "", 0), + } + + g.Add(func() error { + level.Info(logger).Log("msg", "Listening for query and metrics", "address", httpBindAddr) + webHandler.Healthy() + return httpSrv.Serve(listener) + }, func(err error) { + runutil.CloseWithLogOnErr(logger, listener, "receive HTTP listener") + }) } level.Info(logger).Log("msg", "starting receiver") diff --git a/cmd/thanos/rule.go b/cmd/thanos/rule.go index 17bb3273b51..7552211bf54 100644 --- a/cmd/thanos/rule.go +++ b/cmd/thanos/rule.go @@ -29,6 +29,7 @@ import ( "github.com/improbable-eng/thanos/pkg/discovery/dns" "github.com/improbable-eng/thanos/pkg/extprom" "github.com/improbable-eng/thanos/pkg/objstore/client" + "github.com/improbable-eng/thanos/pkg/prober" "github.com/improbable-eng/thanos/pkg/promclient" thanosrule "github.com/improbable-eng/thanos/pkg/rule" v1 "github.com/improbable-eng/thanos/pkg/rule/api" @@ -251,6 +252,8 @@ func runRule( reg.MustRegister(rulesLoaded) reg.MustRegister(ruleEvalWarnings) + readinessProber := prober.NewProber(component.Rule, logger) + for _, addr := range queryAddrs { if addr == "" { return errors.New("static querier address cannot be empty") @@ -517,6 +520,7 @@ func runRule( cancel() }) } + // Start gRPC server. { l, err := net.Listen("tcp", grpcBindAddr) @@ -535,12 +539,15 @@ func runRule( storepb.RegisterStoreServer(s, store) g.Add(func() error { + readinessProber.SetReady() return errors.Wrap(s.Serve(l), "serve gRPC") - }, func(error) { + }, func(err error) { s.Stop() + readinessProber.SetNotReady(err) runutil.CloseWithLogOnErr(logger, l, "store gRPC listener") }) } + // Start UI & metrics HTTP server. { router := route.New() @@ -562,6 +569,8 @@ func runRule( "web.prefix-header": webPrefixHeaderName, } + readinessProber.RegisterInRouter(router) + ui.NewRuleUI(logger, ruleMgrs, alertQueryURL.String(), flagsMap).Register(router.WithPrefix(webRoutePrefix)) api := v1.NewAPI(logger, ruleMgrs) @@ -579,15 +588,17 @@ func runRule( g.Add(func() error { level.Info(logger).Log("msg", "Listening for ui requests", "address", httpBindAddr) + readinessProber.SetHealthy() return errors.Wrap(http.Serve(l, mux), "serve query") - }, func(error) { + }, func(err error) { + readinessProber.SetNotHealthy(err) runutil.CloseWithLogOnErr(logger, l, "query and metric listener") }) } confContentYaml, err := objStoreConfig.Content() if err != nil { - return err + return errors.Wrap(err, "loading flag content") } var uploads = true @@ -601,7 +612,7 @@ func runRule( // new blocks to Google Cloud Storage or an S3-compatible storage service. bkt, err := client.NewBucket(logger, confContentYaml, reg, component.Rule.String()) if err != nil { - return err + return errors.Wrap(err, "initializing bucket") } // Ensure we close up everything properly. diff --git a/cmd/thanos/sidecar.go b/cmd/thanos/sidecar.go index 1e31b54d150..9e869a37b61 100644 --- a/cmd/thanos/sidecar.go +++ b/cmd/thanos/sidecar.go @@ -15,6 +15,7 @@ import ( "github.com/improbable-eng/thanos/pkg/cluster" "github.com/improbable-eng/thanos/pkg/component" "github.com/improbable-eng/thanos/pkg/objstore/client" + "github.com/improbable-eng/thanos/pkg/prober" "github.com/improbable-eng/thanos/pkg/promclient" "github.com/improbable-eng/thanos/pkg/reloader" "github.com/improbable-eng/thanos/pkg/runutil" @@ -111,6 +112,12 @@ func runSidecar( maxt: math.MaxInt64, } + readinessProber := prober.NewProber(component.Sidecar, logger) + err := metricHTTPListenGroup(g, logger, reg, httpBindAddr, *readinessProber) + if err != nil { + return errors.Wrap(err, "create readiness prober") + } + // Setup all the concurrent groups. { promUp := prometheus.NewGauge(prometheus.GaugeOpts{ @@ -133,8 +140,9 @@ func runSidecar( "msg", "failed to fetch initial external labels. Is Prometheus running? Retrying", "err", err, ) + readinessProber.SetNotReady(err) promUp.Set(0) - return err + return errors.Wrap(err, "fetch prometheus external labels") } level.Info(logger).Log( @@ -172,18 +180,21 @@ func runSidecar( if err := m.UpdateLabels(iterCtx, logger); err != nil { level.Warn(logger).Log("msg", "heartbeat failed", "err", err) promUp.Set(0) + readinessProber.SetNotReady(err) } else { // Update gossip. peer.SetLabels(m.LabelsPB()) promUp.Set(1) + readinessProber.SetReady() lastHeartbeat.Set(float64(time.Now().UnixNano()) / 1e9) } return nil }) - }, func(error) { + }, func(err error) { cancel() + readinessProber.SetNotReady(err) peer.Close(2 * time.Second) }) } @@ -191,13 +202,12 @@ func runSidecar( ctx, cancel := context.WithCancel(context.Background()) g.Add(func() error { return reloader.Watch(ctx) - }, func(error) { + }, func(err error) { + readinessProber.SetNotReady(err) cancel() }) } - if err := metricHTTPListenGroup(g, logger, reg, httpBindAddr); err != nil { - return err - } + { l, err := net.Listen("tcp", grpcBindAddr) if err != nil { @@ -222,8 +232,10 @@ func runSidecar( g.Add(func() error { level.Info(logger).Log("msg", "Listening for StoreAPI gRPC", "address", grpcBindAddr) + readinessProber.SetReady() return errors.Wrap(s.Serve(l), "serve gRPC") - }, func(error) { + }, func(err error) { + readinessProber.SetNotReady(err) s.Stop() runutil.CloseWithLogOnErr(logger, l, "store gRPC listener") }) @@ -231,7 +243,7 @@ func runSidecar( confContentYaml, err := objStoreConfig.Content() if err != nil { - return err + return errors.Wrap(err, "loading flag content") } var uploads = true @@ -245,7 +257,7 @@ func runSidecar( // new blocks to Google Cloud Storage or an S3-compatible storage service. bkt, err := client.NewBucket(logger, confContentYaml, reg, component.Sidecar.String()) if err != nil { - return err + return errors.Wrap(err, "initializing bucket") } // Ensure we close up everything properly. diff --git a/cmd/thanos/store.go b/cmd/thanos/store.go index 7f77e15412d..14859d9971f 100644 --- a/cmd/thanos/store.go +++ b/cmd/thanos/store.go @@ -9,16 +9,18 @@ import ( "github.com/go-kit/kit/log" "github.com/go-kit/kit/log/level" "github.com/improbable-eng/thanos/pkg/cluster" + "github.com/improbable-eng/thanos/pkg/component" "github.com/improbable-eng/thanos/pkg/objstore/client" + "github.com/improbable-eng/thanos/pkg/prober" "github.com/improbable-eng/thanos/pkg/runutil" "github.com/improbable-eng/thanos/pkg/store" "github.com/improbable-eng/thanos/pkg/store/storepb" "github.com/oklog/run" - "github.com/opentracing/opentracing-go" + opentracing "github.com/opentracing/opentracing-go" "github.com/pkg/errors" "github.com/prometheus/client_golang/prometheus" "google.golang.org/grpc" - "gopkg.in/alecthomas/kingpin.v2" + kingpin "gopkg.in/alecthomas/kingpin.v2" ) // registerStore registers a store command. @@ -71,7 +73,6 @@ func registerStore(m map[string]setupFunc, app *kingpin.Application, name string uint64(*chunkPoolSize), uint64(*maxSampleCount), int(*maxConcurrent), - name, debugLogging, *syncInterval, *blockSyncConcurrency, @@ -97,18 +98,24 @@ func runStore( chunkPoolSizeBytes uint64, maxSampleCount uint64, maxConcurrent int, - component string, verbose bool, syncInterval time.Duration, blockSyncConcurrency int, ) error { + + readinessProber := prober.NewProber(component.Store, logger) + err := metricHTTPListenGroup(g, logger, reg, httpBindAddr, *readinessProber) + if err != nil { + return errors.Wrap(err, "create readiness prober") + } + { confContentYaml, err := objStoreConfig.Content() if err != nil { return err } - bkt, err := client.NewBucket(logger, confContentYaml, reg, component) + bkt, err := client.NewBucket(logger, confContentYaml, reg, component.Store.String()) if err != nil { return errors.Wrap(err, "create bucket client") } @@ -146,7 +153,6 @@ func runStore( ctx, cancel := context.WithCancel(context.Background()) g.Add(func() error { defer runutil.CloseWithLogOnErr(logger, bkt, "bucket client") - err := runutil.Repeat(syncInterval, ctx.Done(), func() error { if err := bs.SyncBlocks(ctx); err != nil { level.Warn(logger).Log("msg", "syncing blocks failed", "err", err) @@ -176,8 +182,10 @@ func runStore( g.Add(func() error { level.Info(logger).Log("msg", "Listening for StoreAPI gRPC", "address", grpcBindAddr) + readinessProber.SetReady() return errors.Wrap(s.Serve(l), "serve gRPC") - }, func(error) { + }, func(err error) { + readinessProber.SetNotReady(err) runutil.CloseWithLogOnErr(logger, l, "store gRPC listener") }) } @@ -202,9 +210,6 @@ func runStore( peer.Close(5 * time.Second) }) } - if err := metricHTTPListenGroup(g, logger, reg, httpBindAddr); err != nil { - return err - } level.Info(logger).Log("msg", "starting store node") return nil diff --git a/go.sum b/go.sum index 7840ed8f3ae..b7b89b29155 100644 --- a/go.sum +++ b/go.sum @@ -5,11 +5,9 @@ git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGy git.apache.org/thrift.git v0.0.0-20181218151757-9b75e4fe745a/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg= github.com/Azure/azure-pipeline-go v0.1.8 h1:KmVRa8oFMaargVesEuuEoiLCQ4zCCwQ8QX/xg++KS20= github.com/Azure/azure-pipeline-go v0.1.8/go.mod h1:XA1kFWRVhSK+KNFiOhfv83Fv8L9achrP7OxIzeTn1Yg= -github.com/Azure/azure-sdk-for-go v0.0.0-20161028183111-bd73d950fa44 h1:L4fLiifszjLnCRGi6Xhp0MgUwjIMbVXKbayoRiVxkU8= github.com/Azure/azure-sdk-for-go v0.0.0-20161028183111-bd73d950fa44/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc= github.com/Azure/azure-storage-blob-go v0.0.0-20181022225951-5152f14ace1c h1:Y5ueznoCekgCWBytF1Q9lTpZ3tJeX37dQtCcGjMCLYI= github.com/Azure/azure-storage-blob-go v0.0.0-20181022225951-5152f14ace1c/go.mod h1:oGfmITT1V6x//CswqY2gtAHND+xIP64/qL7a5QJix0Y= -github.com/Azure/go-autorest v10.8.1+incompatible h1:u0jVQf+a6k6x8A+sT60l6EY9XZu+kHdnZVPAYqpVRo0= github.com/Azure/go-autorest v10.8.1+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24= github.com/NYTimes/gziphandler v1.1.1 h1:ZUDjpQae29j0ryrS0u/B8HZfJBtBQHjqw2rQ2cqUQ3I= github.com/NYTimes/gziphandler v1.1.1/go.mod h1:n/CVRwUEOgIxrgPvAQhUUr9oeUtvrhMomdKFjzJNB0c= @@ -23,7 +21,6 @@ github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf h1:qet1QNfXsQxTZq github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da h1:8GUt8eRujhVEGZFFEjBj46YV4rDjvGrNxb0KMWYkL2I= github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY= -github.com/aws/aws-sdk-go v0.0.0-20180507225419-00862f899353 h1:qFKf58XUUvHaEz0zFkLJsQ4dzoAyrQ8QyhK4nHGHBI4= github.com/aws/aws-sdk-go v0.0.0-20180507225419-00862f899353/go.mod h1:ZRmQr0FajVIyZ4ZzBYKG5P3ZqPz9IHG41ZoMu1ADI3k= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 h1:xJ4a3vCFaGF/jqvzLMYoU8P317H5OQ+Via4RmuPwCS0= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= @@ -42,13 +39,11 @@ github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/dgrijalva/jwt-go v0.0.0-20161101193935-9ed569b5d1ac h1:xrQJVwQCGqDvOO7/0+RyIq5J2M3Q4ZF7Ug/BMQtML1E= github.com/dgrijalva/jwt-go v0.0.0-20161101193935-9ed569b5d1ac/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= github.com/elastic/gosigar v0.9.0/go.mod h1:cdorVVzy1fhmEqmtgqkoE3bYtCfSCkVyjTyCIo22xvs= github.com/elazarl/go-bindata-assetfs v1.0.0/go.mod h1:v+YaWX3bdea5J/mo8dSETolEo7R71Vk1u8bnjau5yw4= -github.com/evanphx/json-patch v4.1.0+incompatible h1:K1MDoo4AZ4wU0GIU/fPmtZg7VpzLjCxu+UwBD1FvwOc= github.com/evanphx/json-patch v4.1.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/facebookgo/clock v0.0.0-20150410010913-600d898af40a/go.mod h1:7Ga40egUymuWXxAe151lTNnCv97MddSOVsjpPPkityA= github.com/fatih/structtag v1.0.0 h1:pTHj65+u3RKWYPSGaU290FpI/dXxTaHdVwVwbcPKmEc= @@ -59,7 +54,6 @@ github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/getsentry/raven-go v0.1.0/go.mod h1:KungGk8q33+aIAZUIVWZDr2OfAEBsO49PX4NzFV5kcQ= github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= -github.com/go-ini/ini v1.21.1 h1:+QXUYsI7Tfxc64oD6R5BxU/Aq+UwGkyjH4W/hMNG7bg= github.com/go-ini/ini v1.21.1/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8= github.com/go-kit/kit v0.8.0 h1:Wz+5lgoB0kkuqLEc6NVmwRknTKP6dTGbSqvhZtBI/j0= github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= @@ -74,7 +68,6 @@ github.com/gogo/protobuf v1.2.0 h1:xU6/SpYbvkNYiptHJYEDRseDLvYE7wSqhYYNy0QSUzI= github.com/gogo/protobuf v1.2.0/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= -github.com/golang/groupcache v0.0.0-20180924190550-6f2cf27854a4 h1:6UVLWz0fIIrv0UVj6t0A7cL48n8IyAdLVQqAYzEfsKI= github.com/golang/groupcache v0.0.0-20180924190550-6f2cf27854a4/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:tluoj9z5200jBnyusfRPU2LqT6J+DAorxEvtC7LHB+E= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= @@ -84,27 +77,24 @@ github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5y github.com/golang/snappy v0.0.0-20160529050041-d9eb7a3d35ec/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db h1:woRePGFeVFfLKN/pOkfl+p/TAqKOfFu+7KPlMVpok/w= github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= -github.com/google/btree v0.0.0-20180124185431-e89373fe6b4a h1:ZJu5NB1Bk5ms4vw0Xu4i+jD32SE9jQXyfnOvwhHqlT0= github.com/google/btree v0.0.0-20180124185431-e89373fe6b4a/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-querystring v1.0.0 h1:Xkwi/a1rcvNg1PPYe5vI8GbeBY/jrVuDX5ASuANWTrk= github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck= -github.com/google/gofuzz v0.0.0-20150304233714-bbcb9da2d746 h1:M6d2zDTA4cKXT6OwFsJxlo5tWrAukj3KfvJ1zcBatnA= github.com/google/gofuzz v0.0.0-20150304233714-bbcb9da2d746/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI= github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no= github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= +github.com/google/pprof v0.0.0-20180605153948-8b03ce837f34 h1:mGdRet4qWdrDnNidFrlgpa8iNWM/RAwRDEMsLRICCac= github.com/google/pprof v0.0.0-20180605153948-8b03ce837f34/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= github.com/googleapis/gax-go v2.0.2+incompatible h1:silFMLAnr330+NRuag/VjIGF7TLp/LBrV2CJKFLWEww= github.com/googleapis/gax-go v2.0.2+incompatible/go.mod h1:SFVmujtThgffbyetf+mdk2eWhX2bMyUtNHzFKcPA9HY= -github.com/googleapis/gnostic v0.0.0-20180520015035-48a0ecefe2e4 h1:yxHFSapGMUoyn+3v6LiJJxoJhvbDqIq8me0gAWehnSU= github.com/googleapis/gnostic v0.0.0-20180520015035-48a0ecefe2e4/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY= github.com/gophercloud/gophercloud v0.0.0-20181206160319-9d88c34913a9 h1:7TRGugCPfA2Mll6QT7cbhD1GXZwk7+1PUz8tYrOWXgQ= github.com/gophercloud/gophercloud v0.0.0-20181206160319-9d88c34913a9/go.mod h1:3WdhXV3rUYy9p6AUW8d94kr+HS62Y4VL9mBnFxsD8q4= github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= github.com/gopherjs/gopherjs v0.0.0-20181103185306-d547d1d9531e h1:JKmoR8x90Iww1ks85zJ1lfDGgIiMDuIptTOhJq+zKyg= github.com/gopherjs/gopherjs v0.0.0-20181103185306-d547d1d9531e/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= -github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/grpc-ecosystem/go-grpc-middleware v1.0.0 h1:Iju5GlWwrvL6UBg4zJJt3btmonfrMlCDdsejg4CZE7c= github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs= @@ -118,7 +108,6 @@ github.com/hashicorp/consul v0.0.0-20180615161029-bed22a81e9fd h1:auIpcMc3+//R94 github.com/hashicorp/consul v0.0.0-20180615161029-bed22a81e9fd/go.mod h1:mFrjN1mfidgJfYP1xrJCF+AfRhr6Eaqhb2+sfyn/OOI= github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA= github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= -github.com/hashicorp/go-cleanhttp v0.0.0-20160407174126-ad28ea4487f0 h1:2l0haPDqCzZEO160UR5DSrrl8RWptFCoxFsSbRLJBaI= github.com/hashicorp/go-cleanhttp v0.0.0-20160407174126-ad28ea4487f0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= github.com/hashicorp/go-immutable-radix v1.0.0 h1:AKDB1HM5PWEA7i4nhcpwOrO2byshxBjXVn/J/3+z5/0= github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= @@ -126,7 +115,6 @@ github.com/hashicorp/go-msgpack v0.0.0-20150518234257-fa3f63826f7c h1:BTAbnbegUI github.com/hashicorp/go-msgpack v0.0.0-20150518234257-fa3f63826f7c/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM= github.com/hashicorp/go-multierror v1.0.0 h1:iVjPR7a6H0tWELX5NxNe7bYopibicUzc7uPribsnS6o= github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= -github.com/hashicorp/go-rootcerts v0.0.0-20160503143440-6bb64b370b90 h1:VBj0QYQ0u2MCJzBfeYXGexnAl17GsH1yidnoxCqqD9E= github.com/hashicorp/go-rootcerts v0.0.0-20160503143440-6bb64b370b90/go.mod h1:o4zcYY1e0GEZI6eSEr+43QDYmuGglw1qSO6qdHUHCgg= github.com/hashicorp/go-sockaddr v0.0.0-20180320115054-6d291a969b86 h1:7YOlAIO2YWnJZkQp7B5eFykaIY7C9JndqAFQyVV5BhM= github.com/hashicorp/go-sockaddr v0.0.0-20180320115054-6d291a969b86/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerXegt+ozgdvDeDU= @@ -140,20 +128,18 @@ github.com/hashicorp/serf v0.0.0-20161007004122-1d4fa605f6ff h1:3QdMaUEV3zE0VcBg github.com/hashicorp/serf v0.0.0-20161007004122-1d4fa605f6ff/go.mod h1:h/Ru6tmZazX7WO/GDmwdpS975F019L4t5ng5IgwbNrE= github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d h1:kJCB4vdITiW1eC1vq2e6IsrXKrZit1bv/TDYFGMp4BQ= github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d/go.mod h1:+NfK9FKeTrX5uv1uIXGdwYDTeHna2qgaIlx54MXqjAM= -github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= github.com/influxdata/influxdb v0.0.0-20170331210902-15e594fc09f1/go.mod h1:qZna6X/4elxqT3yI9iZYdZrWWdeFOOprn86kgg4+IzY= github.com/jackc/fake v0.0.0-20150926172116-812a484cc733/go.mod h1:WrMFNQdiFJ80sQsxDoMokWK1W5TQtxBFNpzWTD84ibQ= github.com/jackc/pgx v3.2.0+incompatible/go.mod h1:0ZGrqGqkRlliWnWB4zKnWtjbSWbGkVEFm4TeybAXq+I= -github.com/jmespath/go-jmespath v0.0.0-20160803190731-bd40a432e4c7 h1:SMvOWPJCES2GdFracYbBQh93GXac8fq7HeN6JnpduB8= github.com/jmespath/go-jmespath v0.0.0-20160803190731-bd40a432e4c7/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k= -github.com/json-iterator/go v0.0.0-20180612202835-f2b4162afba3 h1:/UewZcckqhvnnS0C6r3Sher2hSEbVmM6Ogpcjen08+Y= github.com/json-iterator/go v0.0.0-20180612202835-f2b4162afba3/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= github.com/jtolds/gls v4.2.1+incompatible h1:fSuqC+Gmlu6l/ZYAoZzx2pyucC8Xza35fpRVWLVmUEE= github.com/jtolds/gls v4.2.1+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= github.com/julienschmidt/httprouter v0.0.0-20150905172533-109e267447e9/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= github.com/julienschmidt/httprouter v1.1.0 h1:7wLdtIiIpzOkC9u6sXOozpBauPdskj3ru4EI5MABq68= github.com/julienschmidt/httprouter v1.1.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= +github.com/kisielk/gotool v1.0.0 h1:AV2c/EiW3KqPNT9ZKl07ehoAGi4C5/01Cfbblndcapg= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/knz/strtime v0.0.0-20181018220328-af2256ee352c/go.mod h1:4ZxfWkxwtc7dBeifERVVWRy9F9rTU9p0yCDgeCtlius= github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515 h1:T+h1c/A9Gawja4Y9mFVWj2vyii2bbUNDw3kt9VxK2EY= @@ -179,12 +165,9 @@ github.com/minio/minio-go v0.0.0-20190131015406-c8a261de75c1/go.mod h1:vuvdOZLJu github.com/mitchellh/go-homedir v0.0.0-20180523094522-3864e76763d9/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y= github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= -github.com/mitchellh/go-testing-interface v1.0.0 h1:fzU/JVNcaqHQEcVFAKeR41fkiLdIPrefOvVG1VZ96U0= github.com/mitchellh/go-testing-interface v1.0.0/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI= github.com/mitchellh/reflectwalk v1.0.0/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw= -github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742 h1:Esafd1046DLDQ0W1YjYsBW+p8U2u7vzgW2SQVmlNazg= github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= github.com/montanaflynn/stats v0.0.0-20180911141734-db72e6cae808/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc= github.com/mozillazg/go-cos v0.11.0 h1:PUaIGDL6Jxgrzqg0XvZ6y32jS/5z26+UxnX6ENMm7yE= @@ -201,9 +184,7 @@ github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn github.com/olekukonko/tablewriter v0.0.0-20180912035003-be2c049b30cc/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo= github.com/olekukonko/tablewriter v0.0.1 h1:b3iUnf1v+ppJiOfNX4yxxqfWKMQPZR5yoh8urCTFX88= github.com/olekukonko/tablewriter v0.0.1/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo= -github.com/onsi/ginkgo v1.6.0 h1:Ix8l273rp3QzYgXSR+c8d1fTG7UPgYkOSELPhiY/YGw= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= -github.com/onsi/gomega v1.4.1 h1:PZSj/UFNaVp3KxrzHOcS7oyuWA7LoOY/77yCTEFu21U= github.com/onsi/gomega v1.4.1/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= github.com/opentracing-contrib/go-stdlib v0.0.0-20170113013457-1de4cc2120e7 h1:8KbikWulLUcMM96hBxjgoo6gTmCkG6HYSDohv/WygYU= github.com/opentracing-contrib/go-stdlib v0.0.0-20170113013457-1de4cc2120e7/go.mod h1:PLldrQSroqzH70Xl+1DQcGnefIbqsKR7UDaiux3zV+w= @@ -216,7 +197,6 @@ github.com/openzipkin/zipkin-go v0.1.1/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTm github.com/openzipkin/zipkin-go v0.1.3/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTmOf0Erfk+hxe8= github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c h1:Lgl0gzECD8GnQ5QCWA8o6BtfL6mDH5rQgM4/fX3avOs= github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= -github.com/peterbourgon/diskv v0.0.0-20180312054125-0646ccaebea1 h1:k/dnb0bixQwWsDLxwr6/w7rtZCVDKdbQnGQkeZGYsws= github.com/peterbourgon/diskv v0.0.0-20180312054125-0646ccaebea1/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= github.com/peterbourgon/g2s v0.0.0-20170223122336-d4e7ad98afea/go.mod h1:1VcHEd3ro4QMoHfiNl/j7Jkln9+KQuorp0PItHMJYNg= github.com/petermattis/goid v0.0.0-20170504144140-0ded85884ba5/go.mod h1:jvVRKCrJTQWu0XVbaOlby/2lO20uSCHEMzzplHXte1o= @@ -248,7 +228,6 @@ github.com/prometheus/tsdb v0.4.0/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40T github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4= github.com/rlmcpherson/s3gof3r v0.5.0/go.mod h1:s7vv7SMDPInkitQMuZzH615G7yWHdrU2r/Go7Bo71Rs= github.com/rubyist/circuitbreaker v2.2.1+incompatible/go.mod h1:Ycs3JgJADPuzJDwffe12k6BZT8hxVi6lFK+gWYJLN4A= -github.com/samuel/go-zookeeper v0.0.0-20161028232340-1d7be4effb13 h1:4AQBn5RJY4WH8t8TLEMZUsWeXHAUcoao42TCAfpEJJE= github.com/samuel/go-zookeeper v0.0.0-20161028232340-1d7be4effb13/go.mod h1:gi+0XIa01GRL2eRQVjQkKGqKF3SF9vZR/HnPullcV2E= github.com/sasha-s/go-deadlock v0.0.0-20161201235124-341000892f3d/go.mod h1:StQn567HiB1fF2yJ44N9au7wOhrPS3iZqiDbRupzT10= github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0= @@ -265,7 +244,6 @@ github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c h1:Ho+uVpke github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c/go.mod h1:XDJAKZRPZ1CvBcN2aX5YOUTYGHki24fSF0Iv48Ibg0s= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72 h1:qLC7fQah7D6K1B0ujays3HV9gkFtllcxhzImRR7ArPQ= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= -github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg= github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= @@ -310,10 +288,10 @@ golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2 h1:z99zHgr7hKfrUcX/KsoJk5FJfjTceCKIp96+biqP4To= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/time v0.0.0-20170424234030-8be79e1e0910 h1:bCMaBn7ph495H+x72gEvgcv+mDRd9dElbzo/mVCMxX4= golang.org/x/time v0.0.0-20170424234030-8be79e1e0910/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20181023010539-40a48ad93fbe/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181219222714-6e267b5cc78e h1:XEcLGV2fKy3FrsoJVCkX+lMhqc9Suj7J5L/wldA1wu4= golang.org/x/tools v0.0.0-20181219222714-6e267b5cc78e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= google.golang.org/api v0.0.0-20180506000402-20530fd5d65a/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= @@ -340,16 +318,13 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8 gopkg.in/check.v1 v1.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= gopkg.in/fsnotify/fsnotify.v1 v1.3.0/go.mod h1:Fyux9zXlo4rWoMSIzpn9fDAYjalPqJ/K1qJ27s+7ltE= gopkg.in/fsnotify/fsnotify.v1 v1.4.7 h1:XNNYLJHt73EyYiCZi6+xjupS9CpvmiDgjPTAjrBlQbo= gopkg.in/fsnotify/fsnotify.v1 v1.4.7/go.mod h1:Fyux9zXlo4rWoMSIzpn9fDAYjalPqJ/K1qJ27s+7ltE= -gopkg.in/inf.v0 v0.9.0 h1:3zYtXIO92bvsdS3ggAdA8Gb4Azj0YU+TVY1uGYNFA8o= gopkg.in/inf.v0 v0.9.0/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= gopkg.in/ini.v1 v1.41.0 h1:Ka3ViY6gNYSKiVy71zXBEqKplnV35ImDLVG+8uoIklE= gopkg.in/ini.v1 v1.41.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= -gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= gopkg.in/vmihailenco/msgpack.v2 v2.9.1 h1:kb0VV7NuIojvRfzwslQeP3yArBqJHW9tOl4t38VS1jM= gopkg.in/vmihailenco/msgpack.v2 v2.9.1/go.mod h1:/3Dn1Npt9+MYyLpYYXjInO/5jvMLamn+AEGwNEOatn8= @@ -358,20 +333,15 @@ gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20180920025451-e3ad64cb4ed3 h1:LyX67rVB0kBUFoROrQfzKwdrYLH1cRzHibxdJW85J1c= honnef.co/go/tools v0.0.0-20180920025451-e3ad64cb4ed3/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= -k8s.io/api v0.0.0-20181213150558-05914d821849 h1:WZFcFPXmLR7g5CxQNmjWv0mg8qulJLxDghbzS4pQtzY= k8s.io/api v0.0.0-20181213150558-05914d821849/go.mod h1:iuAfoD4hCxJ8Onx9kaTIt30j7jUFS00AXQi6QMi99vA= -k8s.io/apimachinery v0.0.0-20181127025237-2b1284ed4c93 h1:tT6oQBi0qwLbbZSfDkdIsb23EwaLY85hoAV4SpXfdao= k8s.io/apimachinery v0.0.0-20181127025237-2b1284ed4c93/go.mod h1:ccL7Eh7zubPUSh9A3USN90/OzHNSVN6zxzde07TDCL0= -k8s.io/client-go v2.0.0-alpha.0.0.20181121191925-a47917edff34+incompatible h1:7JnS1I1KbtbearjSCrycUhHSob+KjG6HDWY1GhjkAIU= k8s.io/client-go v2.0.0-alpha.0.0.20181121191925-a47917edff34+incompatible/go.mod h1:7vJpHMYJwNQCWgzmNV+VYUl1zCObLyodBc8nIyt8L5s= -k8s.io/klog v0.1.0 h1:I5HMfc/DtuVaGR1KPwUrTc476K8NCqNBldC7H4dYEzk= k8s.io/klog v0.1.0/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk= -k8s.io/kube-openapi v0.0.0-20180629012420-d83b052f768a h1:tHgpQvrWaYfrnC8G4N0Oszw5HHCsZxKilDi2R7HuCSM= k8s.io/kube-openapi v0.0.0-20180629012420-d83b052f768a/go.mod h1:BXM9ceUBTj2QnfH2MK1odQs778ajze1RxcmP6S8RVVc= labix.org/v2/mgo v0.0.0-20140701140051-000000000287 h1:L0cnkNl4TfAXzvdrqsYEmxOHOCv2p5I3taaReO8BWFs= labix.org/v2/mgo v0.0.0-20140701140051-000000000287/go.mod h1:Lg7AYkt1uXJoR9oeSZ3W/8IXLdvOfIITgZnommstyz4= launchpad.net/gocheck v0.0.0-20140225173054-000000000087 h1:Izowp2XBH6Ya6rv+hqbceQyw/gSGoXfH/UPoTGduL54= launchpad.net/gocheck v0.0.0-20140225173054-000000000087/go.mod h1:hj7XX3B/0A+80Vse0e+BUHsHMTEhd0O4cpUHr/e/BUM= -sigs.k8s.io/yaml v1.1.0 h1:4A07+ZFc2wgJwo8YNlQpr1rVlgUDlxXHhPJciaPY5gs= sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o= diff --git a/pkg/component/component.go b/pkg/component/component.go index 690afa1ba5b..84b2e359f07 100644 --- a/pkg/component/component.go +++ b/pkg/component/component.go @@ -6,6 +6,11 @@ import ( "github.com/improbable-eng/thanos/pkg/store/storepb" ) +// Component is a generic component interface +type Component interface { + String() string +} + // StoreAPI is a component that implements Thanos' gRPC StoreAPI. type StoreAPI interface { implementsStoreAPI() diff --git a/pkg/prober/prober.go b/pkg/prober/prober.go new file mode 100644 index 00000000000..b9d32ec6aa0 --- /dev/null +++ b/pkg/prober/prober.go @@ -0,0 +1,157 @@ +package prober + +import ( + "fmt" + "io" + "net/http" + "sync" + + "github.com/go-kit/kit/log" + "github.com/go-kit/kit/log/level" + "github.com/improbable-eng/thanos/pkg/component" + "github.com/prometheus/common/route" +) + +const ( + healthyEndpointPath = "/-/healthy" + readyEndpointPath = "/-/ready" + okProbeText = "thanos %v is %v" + errorProbeText = "thanos %v is not %v. Reason: %v" + probeErrorHTTPStatus = 503 + initialErrorText = "thanos %s is initializing" +) + +// Prober represents health and readriness status of given compoent. +type Prober struct { + logger log.Logger + componentMtx sync.Mutex + component component.Component + readyMtx sync.Mutex + readiness error + healthyMtx sync.Mutex + healthiness error +} + +// SetLogger sets logger used by the Prober. +func (p *Prober) SetLogger(logger log.Logger) { + p.logger = logger +} + +func (p *Prober) getLogger() log.Logger { + return p.logger +} + +// SetComponent sets component name of the Prober displayed in responses. +func (p *Prober) SetComponent(component component.Component) { + p.componentMtx.Lock() + defer p.componentMtx.Unlock() + p.component = component +} + +func (p *Prober) getComponent() component.Component { + p.componentMtx.Lock() + defer p.componentMtx.Unlock() + return p.component +} + +// NewProber returns Prober reprezenting readi ness and healthiness of given component. +func NewProber(component component.Component, logger log.Logger) *Prober { + initialErr := fmt.Errorf(initialErrorText, component) + prober := &Prober{} + prober.SetComponent(component) + prober.SetLogger(logger) + prober.SetNotHealthy(initialErr) + prober.SetNotReady(initialErr) + return prober +} + +// HandleInMux registers readiness and liveness probes to mux. +func (p *Prober) RegisterInRouter(router *route.Router) { + router.Get(healthyEndpointPath, p.probeHandlerFunc(p.IsHealthy, "healthy")) + router.Get(readyEndpointPath, p.probeHandlerFunc(p.IsReady, "ready")) +} + +// RegisterInMux registers readiness and liveness probes to mux. +func (p *Prober) RegisterInMux(mux *http.ServeMux) { + mux.HandleFunc(healthyEndpointPath, p.probeHandlerFunc(p.IsHealthy, "healthy")) + mux.HandleFunc(readyEndpointPath, p.probeHandlerFunc(p.IsReady, "ready")) +} + +func (p *Prober) writeResponse(w http.ResponseWriter, probeFunc func() error, probeType string) { + err := probeFunc() + if err != nil { + http.Error(w, fmt.Sprintf(errorProbeText, p.getComponent(), probeType, err), probeErrorHTTPStatus) + return + } + if _, e := io.WriteString(w, fmt.Sprintf(okProbeText, p.getComponent(), probeType)); e == nil { + level.Error(p.getLogger()).Log("msg", "failed to write probe response", "probe type", probeType, "err", err) + } +} + +func (p *Prober) probeHandlerFunc(probeFunc func() error, probeType string) func(http.ResponseWriter, *http.Request) { + return func(w http.ResponseWriter, _ *http.Request) { + p.writeResponse(w, probeFunc, probeType) + } +} + +// IsReady returns error if component is not ready and nil if it is. +func (p *Prober) IsReady() error { + p.readyMtx.Lock() + defer p.readyMtx.Unlock() + return p.readiness +} + +// SetReady sets components status to ready. +func (p *Prober) SetReady() { + if p.IsReady() != nil { + level.Info(p.getLogger()).Log("msg", "changing probe status", "status", "ready") + } + p.SetNotReady(nil) +} + +// SetNotReady sets components status to not ready with given error as a cause. +func (p *Prober) SetNotReady(err error) { + p.readyMtx.Lock() + defer p.readyMtx.Unlock() + if err != nil && p.IsReady() == nil { + level.Warn(p.getLogger()).Log("msg", "changing probe status", "status", "not-ready", "reason", err) + } + p.readiness = err +} + +// IsHealthy returns error if component is not healthy and nil if it is. +func (p *Prober) IsHealthy() error { + p.healthyMtx.Lock() + defer p.healthyMtx.Unlock() + return p.healthiness +} + +// SetHealthy sets components status to healthy. +func (p *Prober) SetHealthy() { + if p.IsHealthy() != nil { + level.Info(p.getLogger()).Log("msg", "changing probe status", "status", "healthy") + } + p.SetNotHealthy(nil) +} + +// SetNotHealthy sets components status to not healthy with given error as a cause. +func (p *Prober) SetNotHealthy(err error) { + p.healthyMtx.Lock() + defer p.healthyMtx.Unlock() + if err != nil && p.IsHealthy() == nil { + level.Warn(p.getLogger()).Log("msg", "changing probe status", "status", "unhealthy", "reason", err) + } + p.healthiness = err +} + +// HandleIfReady if probe is ready calls the function otherwise returns 503. +func (p *Prober) HandleIfReady(f http.HandlerFunc) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + ready_err := p.IsReady() + if ready_err == nil { + f(w, r) + return + } + p.writeResponse(w, p.IsReady, "ready") + } +} diff --git a/pkg/prober/prober_test.go b/pkg/prober/prober_test.go new file mode 100644 index 00000000000..4cbd05510ed --- /dev/null +++ b/pkg/prober/prober_test.go @@ -0,0 +1,172 @@ +package prober + +import ( + "context" + "fmt" + "net" + "net/http" + "path" + "testing" + "time" + + "github.com/go-kit/kit/log" + "github.com/improbable-eng/thanos/pkg/runutil" + "github.com/improbable-eng/thanos/pkg/testutil" + "github.com/oklog/run" + "github.com/pkg/errors" + "github.com/prometheus/common/route" +) + +func queryHTTPGetEndpoint(ctx context.Context, t *testing.T, logger log.Logger, url string) (*http.Response, error) { + req, err := http.NewRequest("GET", fmt.Sprintf("http://%s", url), nil) + testutil.Ok(t, err) + return http.DefaultClient.Do(req.WithContext(ctx)) +} + +type TestComponent struct { + name string +} + +func (c TestComponent) String() string { + return c.name +} + +func TestProberHealthInitialState(t *testing.T) { + component := TestComponent{name: "test"} + expectedErrorMessage := fmt.Sprintf(initialErrorText, component) + p := NewProber(component, log.NewNopLogger()) + + err := p.IsHealthy() + testutil.NotOk(t, err) + testutil.Equals(t, err.Error(), expectedErrorMessage) +} + +func TestProberReadinessInitialState(t *testing.T) { + component := TestComponent{name: "test"} + expectedErrorMessage := fmt.Sprintf(initialErrorText, component) + p := NewProber(component, log.NewNopLogger()) + + err := p.IsReady() + testutil.NotOk(t, err) + testutil.Equals(t, err.Error(), expectedErrorMessage) +} + +func TestProberReadyStatusSetting(t *testing.T) { + component := TestComponent{name: "test"} + testError := fmt.Errorf("test error") + p := NewProber(component, log.NewNopLogger()) + + p.SetReady() + err := p.IsReady() + testutil.Equals(t, err, nil) + p.SetNotReady(testError) + err = p.IsReady() + testutil.NotOk(t, err) +} + +func TestProberHeatlthyStatusSetting(t *testing.T) { + component := TestComponent{name: "test"} + testError := fmt.Errorf("test error") + p := NewProber(component, log.NewNopLogger()) + + p.SetHealthy() + err := p.IsHealthy() + testutil.Equals(t, err, nil) + p.SetNotHealthy(testError) + err = p.IsHealthy() + testutil.NotOk(t, err) +} + +func TestProberMuxRegistering(t *testing.T) { + component := TestComponent{name: "test"} + ctx := context.Background() + var g run.Group + mux := http.NewServeMux() + + freePort, err := testutil.FreePort() + testutil.Ok(t, err) + serverAddress := fmt.Sprintf("localhost:%d", freePort) + + l, err := net.Listen("tcp", serverAddress) + testutil.Ok(t, err) + g.Add(func() error { + return errors.Wrap(http.Serve(l, mux), "serve probes") + }, func(error) {}) + + p := NewProber(component, log.NewNopLogger()) + p.RegisterInMux(mux) + + go func() { _ = g.Run() }() + + testutil.Ok(t, runutil.Retry(time.Second, ctx.Done(), func() error { + resp, err := queryHTTPGetEndpoint(ctx, t, log.NewNopLogger(), path.Join(serverAddress, healthyEndpointPath)) + testutil.Ok(t, err) + testutil.Equals(t, probeErrorHTTPStatus, resp.StatusCode) + return err + })) + + testutil.Ok(t, runutil.Retry(time.Second, ctx.Done(), func() error { + resp, err := queryHTTPGetEndpoint(ctx, t, log.NewNopLogger(), path.Join(serverAddress, readyEndpointPath)) + testutil.Ok(t, err) + testutil.Equals(t, probeErrorHTTPStatus, resp.StatusCode) + return err + })) + + p.SetHealthy() + + testutil.Ok(t, runutil.Retry(time.Second, ctx.Done(), func() error { + resp, err := queryHTTPGetEndpoint(ctx, t, log.NewNopLogger(), path.Join(serverAddress, healthyEndpointPath)) + testutil.Ok(t, err) + testutil.Equals(t, 200, resp.StatusCode) + return err + })) + +} + +func TestProberRouterRegistering(t *testing.T) { + component := TestComponent{name: "test"} + router := route.New() + ctx := context.Background() + var g run.Group + mux := http.NewServeMux() + + freePort, err := testutil.FreePort() + testutil.Ok(t, err) + serverAddress := fmt.Sprintf("localhost:%d", freePort) + + l, err := net.Listen("tcp", serverAddress) + testutil.Ok(t, err) + g.Add(func() error { + return errors.Wrap(http.Serve(l, mux), "serve probes") + }, func(error) {}) + + p := NewProber(component, log.NewNopLogger()) + p.RegisterInRouter(router) + mux.Handle("/", router) + + go func() { _ = g.Run() }() + + testutil.Ok(t, runutil.Retry(time.Second, ctx.Done(), func() error { + resp, err := queryHTTPGetEndpoint(ctx, t, log.NewNopLogger(), path.Join(serverAddress, healthyEndpointPath)) + testutil.Ok(t, err) + testutil.Equals(t, probeErrorHTTPStatus, resp.StatusCode) + return err + })) + + testutil.Ok(t, runutil.Retry(time.Second, ctx.Done(), func() error { + resp, err := queryHTTPGetEndpoint(ctx, t, log.NewNopLogger(), path.Join(serverAddress, readyEndpointPath)) + testutil.Ok(t, err) + testutil.Equals(t, probeErrorHTTPStatus, resp.StatusCode) + return err + })) + + p.SetHealthy() + + testutil.Ok(t, runutil.Retry(time.Second, ctx.Done(), func() error { + resp, err := queryHTTPGetEndpoint(ctx, t, log.NewNopLogger(), path.Join(serverAddress, healthyEndpointPath)) + testutil.Ok(t, err) + testutil.Equals(t, 200, resp.StatusCode) + return err + })) + +} diff --git a/pkg/receive/handler.go b/pkg/receive/handler.go index 2ca1f14ce1a..c3d5d1dfd21 100644 --- a/pkg/receive/handler.go +++ b/pkg/receive/handler.go @@ -1,24 +1,15 @@ package receive import ( - "context" - "fmt" "io/ioutil" - stdlog "log" - "net" "net/http" - "sync/atomic" "github.com/go-kit/kit/log" "github.com/go-kit/kit/log/level" "github.com/gogo/protobuf/proto" "github.com/golang/snappy" - "github.com/improbable-eng/thanos/pkg/runutil" + "github.com/improbable-eng/thanos/pkg/prober" "github.com/improbable-eng/thanos/pkg/store/prompb" - conntrack "github.com/mwitkow/go-conntrack" - "github.com/oklog/run" - "github.com/opentracing-contrib/go-stdlib/nethttp" - opentracing "github.com/opentracing/opentracing-go" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp" "github.com/prometheus/common/route" @@ -46,22 +37,21 @@ var ( // Options for the web Handler. type Options struct { - Receiver *Writer - ListenAddress string - Registry prometheus.Registerer - ReadyStorage *promtsdb.ReadyStorage + Receiver *Writer + ReadinessProber *prober.Prober + Registry prometheus.Registerer + ReadyStorage *promtsdb.ReadyStorage } // Handler serves a Prometheus remote write receiving HTTP endpoint. type Handler struct { - readyStorage *promtsdb.ReadyStorage - logger log.Logger - receiver *Writer - router *route.Router - options *Options - quitCh chan struct{} - - ready uint32 // ready is uint32 rather than boolean to be able to use atomic functions. + readyStorage *promtsdb.ReadyStorage + logger log.Logger + receiver *Writer + router *route.Router + options *Options + quitCh chan struct{} + readinessProber *prober.Prober } func instrumentHandler(handlerName string, handler http.HandlerFunc) http.HandlerFunc { @@ -81,45 +71,42 @@ func NewHandler(logger log.Logger, o *Options) *Handler { } h := &Handler{ - logger: logger, - router: router, - readyStorage: o.ReadyStorage, - receiver: o.Receiver, - options: o, - quitCh: make(chan struct{}), + logger: logger, + router: router, + readyStorage: o.ReadyStorage, + receiver: o.Receiver, + options: o, + quitCh: make(chan struct{}), + readinessProber: o.ReadinessProber, } - readyf := h.testReady - router.Post("/api/v1/receive", readyf(h.receive)) + router.Post("/api/v1/receive", h.readinessProber.HandleIfReady(h.receive)) + o.Registry.MustRegister( + requestDuration, + responseSize, + ) return h } +// Ready sets Handler to be healthy. +func (h *Handler) Healthy() { + h.readinessProber.SetHealthy() +} + // Ready sets Handler to be ready. func (h *Handler) Ready() { - atomic.StoreUint32(&h.ready, 1) + h.readinessProber.SetReady() } // Verifies whether the server is ready or not. -func (h *Handler) isReady() bool { - ready := atomic.LoadUint32(&h.ready) - return ready > 0 +func (h *Handler) isReady() error { + return h.readinessProber.IsReady() } -// Checks if server is ready, calls f if it is, returns 503 if it is not. -func (h *Handler) testReady(f http.HandlerFunc) http.HandlerFunc { - return func(w http.ResponseWriter, r *http.Request) { - if h.isReady() { - f(w, r) - return - } - - w.WriteHeader(http.StatusServiceUnavailable) - _, err := fmt.Fprintf(w, "Service Unavailable") - if err != nil { - h.logger.Log("msg", "failed to write to response body", "err", err) - } - } +// HandleInMux hadles this router in specified mux on given part +func (h *Handler) HandleInMux(path string, mux *http.ServeMux) { + mux.Handle(path, h.router) } // Quit returns the receive-only quit channel. @@ -127,48 +114,6 @@ func (h *Handler) Quit() <-chan struct{} { return h.quitCh } -// Checks if server is ready, calls f if it is, returns 503 if it is not. -func (h *Handler) testReadyHandler(f http.Handler) http.HandlerFunc { - return h.testReady(f.ServeHTTP) -} - -// Run serves the HTTP endpoints. -func (h *Handler) Run(ctx context.Context) error { - level.Info(h.logger).Log("msg", "Start listening for connections", "address", h.options.ListenAddress) - - listener, err := net.Listen("tcp", h.options.ListenAddress) - if err != nil { - return err - } - - // Monitor incoming connections with conntrack. - listener = conntrack.NewListener(listener, - conntrack.TrackWithName("http"), - conntrack.TrackWithTracing()) - - operationName := nethttp.OperationNameFunc(func(r *http.Request) string { - return fmt.Sprintf("%s %s", r.Method, r.URL.Path) - }) - mux := http.NewServeMux() - mux.Handle("/", h.router) - - errlog := stdlog.New(log.NewStdlibAdapter(level.Error(h.logger)), "", 0) - - httpSrv := &http.Server{ - Handler: nethttp.Middleware(opentracing.GlobalTracer(), mux, operationName), - ErrorLog: errlog, - } - - var g run.Group - g.Add(func() error { - return httpSrv.Serve(listener) - }, func(error) { - runutil.CloseWithLogOnErr(h.logger, listener, "receive HTTP listener") - }) - - return g.Run() -} - func (h *Handler) receive(w http.ResponseWriter, req *http.Request) { compressed, err := ioutil.ReadAll(req.Body) if err != nil { diff --git a/scripts/quickstart.sh b/scripts/quickstart.sh index 334d73f2dab..62fbc446052 100755 --- a/scripts/quickstart.sh +++ b/scripts/quickstart.sh @@ -139,8 +139,7 @@ then --log.level debug \ --tsdb.path "./data/remote-write-receive-data" \ --grpc-address 0.0.0.0:19891 \ - --http-address 0.0.0.0:19691 \ - --remote-write.address 0.0.0.0:19291 & + --http-address 0.0.0.0:19691 & mkdir -p "data/local-prometheus-data/" cat < data/local-prometheus-data/prometheus.yml diff --git a/test/e2e/spinup_test.go b/test/e2e/spinup_test.go index 34bd40f6954..f344a480f07 100644 --- a/test/e2e/spinup_test.go +++ b/test/e2e/spinup_test.go @@ -39,9 +39,8 @@ var ( rulerGRPC = func(i int) string { return fmt.Sprintf("127.0.0.1:%d", 19790+i) } rulerHTTP = func(i int) string { return fmt.Sprintf("127.0.0.1:%d", 19890+i) } - remoteWriteReceiveHTTP = func(i int) string { return fmt.Sprintf("127.0.0.1:%d", 18690+i) } - remoteWriteReceiveGRPC = func(i int) string { return fmt.Sprintf("127.0.0.1:%d", 18790+i) } - remoteWriteReceiveMetricHTTP = func(i int) string { return fmt.Sprintf("127.0.0.1:%d", 18890+i) } + remoteWriteReceiveGRPC = func(i int) string { return fmt.Sprintf("127.0.0.1:%d", 18790+i) } + remoteWriteReceiveHTTP = func(i int) string { return fmt.Sprintf("127.0.0.1:%d", 18890+i) } storeGatewayGRPC = func(i int) string { return fmt.Sprintf("127.0.0.1:%d", 20090+i) } storeGatewayHTTP = func(i int) string { return fmt.Sprintf("127.0.0.1:%d", 20190+i) } @@ -143,8 +142,7 @@ func receiver(i int, config string) cmdScheduleFunc { return append(cmds, newCmdExec(exec.Command("thanos", "receive", "--debug.name", fmt.Sprintf("remote-write-receive-%d", i), "--grpc-address", remoteWriteReceiveGRPC(i), - "--http-address", remoteWriteReceiveMetricHTTP(i), - "--remote-write.address", remoteWriteReceiveHTTP(i), + "--http-address", remoteWriteReceiveHTTP(i), "--tsdb.path", promDir, "--log.level", "debug"))), nil } diff --git a/tutorials/kubernetes-demo/manifests/prometheus-ha-sidecar-lts.yaml b/tutorials/kubernetes-demo/manifests/prometheus-ha-sidecar-lts.yaml index 02d37d2dc5f..2065acf542b 100644 --- a/tutorials/kubernetes-demo/manifests/prometheus-ha-sidecar-lts.yaml +++ b/tutorials/kubernetes-demo/manifests/prometheus-ha-sidecar-lts.yaml @@ -145,6 +145,14 @@ spec: containerPort: 10902 - name: grpc containerPort: 10901 + livenessProbe: + httpGet: + path: /-/healthy + port: http-sidecar + readinessProbe: + httpGet: + path: /-/ready + port: http-sidecar volumeMounts: - name: prometheus mountPath: /var/prometheus diff --git a/tutorials/kubernetes-demo/manifests/prometheus-ha-sidecar.yaml b/tutorials/kubernetes-demo/manifests/prometheus-ha-sidecar.yaml index 9e30120adec..2d7ead968d7 100644 --- a/tutorials/kubernetes-demo/manifests/prometheus-ha-sidecar.yaml +++ b/tutorials/kubernetes-demo/manifests/prometheus-ha-sidecar.yaml @@ -129,6 +129,14 @@ spec: containerPort: 10902 - name: grpc containerPort: 10901 + livenessProbe: + httpGet: + path: /-/healthy + port: http-sidecar + readinessProbe: + httpGet: + path: /-/ready + port: http-sidecar volumeMounts: - name: prometheus mountPath: /var/prometheus diff --git a/tutorials/kubernetes-demo/manifests/thanos-compactor.yaml b/tutorials/kubernetes-demo/manifests/thanos-compactor.yaml index f3174b00a6f..f9d63d3ec52 100644 --- a/tutorials/kubernetes-demo/manifests/thanos-compactor.yaml +++ b/tutorials/kubernetes-demo/manifests/thanos-compactor.yaml @@ -35,6 +35,14 @@ spec: ports: - name: http containerPort: 10902 + livenessProbe: + httpGet: + path: /-/healthy + port: http + readinessProbe: + httpGet: + path: /-/ready + port: http resources: limits: cpu: "1" diff --git a/tutorials/kubernetes-demo/manifests/thanos-querier-no-us1.yaml b/tutorials/kubernetes-demo/manifests/thanos-querier-no-us1.yaml index e3d6f4d5e36..c3ab1c3abd4 100644 --- a/tutorials/kubernetes-demo/manifests/thanos-querier-no-us1.yaml +++ b/tutorials/kubernetes-demo/manifests/thanos-querier-no-us1.yaml @@ -49,6 +49,10 @@ spec: httpGet: path: /-/healthy port: http + readinessProbe: + httpGet: + path: /-/ready + port: http --- apiVersion: v1 kind: Service diff --git a/tutorials/kubernetes-demo/manifests/thanos-querier.yaml b/tutorials/kubernetes-demo/manifests/thanos-querier.yaml index 5cbb3d97428..e8a3e0a623d 100644 --- a/tutorials/kubernetes-demo/manifests/thanos-querier.yaml +++ b/tutorials/kubernetes-demo/manifests/thanos-querier.yaml @@ -52,6 +52,10 @@ spec: httpGet: path: /-/healthy port: http + readinessProbe: + httpGet: + path: /-/ready + port: http --- apiVersion: v1 kind: Service diff --git a/tutorials/kubernetes-demo/manifests/thanos-ruler.yaml b/tutorials/kubernetes-demo/manifests/thanos-ruler.yaml index 8f90c022944..68403a29e89 100644 --- a/tutorials/kubernetes-demo/manifests/thanos-ruler.yaml +++ b/tutorials/kubernetes-demo/manifests/thanos-ruler.yaml @@ -76,6 +76,14 @@ spec: containerPort: 10902 - name: grpc containerPort: 10901 + livenessProbe: + httpGet: + path: /-/healthy + port: http + readinessProbe: + httpGet: + path: /-/ready + port: http resources: limits: cpu: 500m @@ -105,4 +113,4 @@ spec: name: http selector: statefulset.kubernetes.io/pod-name: thanos-ruler-0 - type: NodePort \ No newline at end of file + type: NodePort diff --git a/tutorials/kubernetes-demo/manifests/thanos-store-gateway.yaml b/tutorials/kubernetes-demo/manifests/thanos-store-gateway.yaml index c080cf58c0e..fa60f3fdeb8 100644 --- a/tutorials/kubernetes-demo/manifests/thanos-store-gateway.yaml +++ b/tutorials/kubernetes-demo/manifests/thanos-store-gateway.yaml @@ -40,6 +40,14 @@ spec: containerPort: 10902 - name: grpc containerPort: 10901 + livenessProbe: + httpGet: + path: /-/healthy + port: http + readinessProbe: + httpGet: + path: /-/ready + port: http resources: limits: cpu: "1"