grpc port does not bind as specified via --grpc-address

[storedcc]

Thanos, Prometheus and Golang version used:

thanos, version 0.17.2 (branch: HEAD, revision: 37e6ef6)
build user: root@92283ccb0bc0
build date: 20201208-10:00:57
go version: go1.15
platform: linux/amd64

Object Storage Provider: minio / s3

What happened:
the thanos store component does not bind to the specified port I entered in the cli parameters:
/usr/local/bin/thanos-store store --grpc-address=0.0.0.0:10901 --http-address=0.0.0.0:10902 --data-dir=/var/lib/thanos/store --objstore.config-file=/etc/thanos-store/bucket_config.yaml --log.level=debug

Results in the port 10901 not being binded (saw it working before). This results in the query frontend being unable to gather metrics from the store gateway, as it only receives:
curl: (7) Failed connect to >******<; Connection refused

What you expected to happen:
GRPC interface of the storegw working as expected and exposing the interface to the query component

How to reproduce it (as minimally and precisely as possible):
Start with the mentioned version and the mentioned cli parameters, shouldn't work out of the box

Full logs to relevant components:

Dec 18 14:51:52 store-01.poc-cluster.monit.nitrado.net systemd[1]: Started Thanos Store.
Dec 18 14:51:52 store-01.poc-cluster.monit.nitrado.net thanos-store[17162]: level=debug ts=2020-12-18T13:51:52.616674572Z caller=main.go:64 msg="maxprocs: Leaving GOMAXPROCS=[4]: CPU quota undefined"
Dec 18 14:51:52 store-01.poc-cluster.monit.nitrado.net thanos-store[17162]: level=info ts=2020-12-18T13:51:52.617193564Z caller=main.go:98 msg="Tracing will be disabled"
Dec 18 14:51:52 store-01.poc-cluster.monit.nitrado.net thanos-store[17162]: level=info ts=2020-12-18T13:51:52.617497682Z caller=factory.go:46 msg="loading bucket configuration"
Dec 18 14:51:52 store-01.poc-cluster.monit.nitrado.net thanos-store[17162]: level=info ts=2020-12-18T13:51:52.618860264Z caller=inmemory.go:172 msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
Dec 18 14:51:52 store-01.poc-cluster.monit.nitrado.net thanos-store[17162]: level=info ts=2020-12-18T13:51:52.619746472Z caller=options.go:23 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
Dec 18 14:51:52 store-01.poc-cluster.monit.nitrado.net thanos-store[17162]: level=info ts=2020-12-18T13:51:52.621855183Z caller=store.go:394 msg="starting store node"
Dec 18 14:51:52 store-01.poc-cluster.monit.nitrado.net thanos-store[17162]: level=info ts=2020-12-18T13:51:52.622055119Z caller=store.go:326 msg="initializing bucket store"
Dec 18 14:51:52 store-01.poc-cluster.monit.nitrado.net thanos-store[17162]: level=info ts=2020-12-18T13:51:52.622694063Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
Dec 18 14:51:52 store-01.poc-cluster.monit.nitrado.net thanos-store[17162]: level=info ts=2020-12-18T13:51:52.622745069Z caller=http.go:58 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902

Environment:

• OS (e.g. from /etc/os-release): CentOS Linux release 7.9.2009 (Core)
• Kernel (e.g. uname -a): Linux 3.10.0-693.11.6.el7.x86_64 Initial structure and block shipper #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[BradErz]

Just trying to help here @storedcc

I assume for some reason your block store is not being initialized? That has to happen before the grpc api is actually started.

What is the size of the folder /var/lib/thanos/store?
How much data is inside the bucket?

Just some example files which i used to get the below output:

1. Start minio

docker run -d --rm --name minio \                                                                                  
     -v /home/bwilsonhunt/minio-data:/data \
     -p 9000:9000 -e "MINIO_ACCESS_KEY=minio" -e "MINIO_SECRET_KEY=melovethanos" \
     minio/minio \
     server /data

2. Create objstore.conf file for thanos-store.

type: S3
config:
  bucket: "thanos"
  endpoint: "127.0.0.1:9000"
  insecure: true
  signature_version2: true
  access_key: "minio"
  secret_key: "melovethanos"

3. Run the store with that conf file.

➜  ~ docker run --net=host -v /home/bwilsonhunt/objstore.conf:/etc/objstore.conf quay.io/thanos/thanos:v0.17.2 store --grpc-address=0.0.0.0:10901 --http-address=0.0.0.0:10902 --objstore.config-file=/etc/objstore.conf --log.level=debug 
level=debug ts=2021-01-12T21:50:37.109234052Z caller=main.go:64 msg="maxprocs: Leaving GOMAXPROCS=[8]: CPU quota undefined"
level=info ts=2021-01-12T21:50:37.109362693Z caller=main.go:98 msg="Tracing will be disabled"
level=info ts=2021-01-12T21:50:37.109504235Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2021-01-12T21:50:37.109811554Z caller=inmemory.go:172 msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
level=info ts=2021-01-12T21:50:37.110632722Z caller=options.go:23 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2021-01-12T21:50:37.111474994Z caller=store.go:394 msg="starting store node"
level=info ts=2021-01-12T21:50:37.111630725Z caller=store.go:326 msg="initializing bucket store"
level=info ts=2021-01-12T21:50:37.111755962Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2021-01-12T21:50:37.111787192Z caller=http.go:58 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2021-01-12T21:50:37.114801294Z caller=fetcher.go:458 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=3.084629ms cached=0 returned=0 partial=0
level=info ts=2021-01-12T21:50:37.115083689Z caller=store.go:332 msg="bucket store ready" init_duration=3.368546ms
level=info ts=2021-01-12T21:50:37.115185475Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2021-01-12T21:50:37.115237607Z caller=grpc.go:116 service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10901
level=info ts=2021-01-12T21:50:37.115885267Z caller=fetcher.go:458 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=782.071µs cached=0 returned=0 partial=0

If you compare the output of our logs your bucket store is initialized but never becomes ready. I assume for whatever reason yours gets stuck before

level=info ts=2021-01-12T21:50:37.114801294Z caller=fetcher.go:458 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=3.084629ms cached=0 returned=0 partial=0

Actually if i stop minio from running and just open the port with netcat i get the same behaviour as you.

nc -l 9000

And then restart the thanos-store

➜  ~ docker run --net=host -v /home/bwilsonhunt/objstore.conf:/etc/objstore.conf quay.io/thanos/thanos:v0.17.2 store --grpc-address=0.0.0.0:10901 --http-address=0.0.0.0:10902 --objstore.config-file=/etc/objstore.conf --log.level=debug
level=debug ts=2021-01-12T22:00:53.759423548Z caller=main.go:64 msg="maxprocs: Leaving GOMAXPROCS=[8]: CPU quota undefined"
level=info ts=2021-01-12T22:00:53.759574837Z caller=main.go:98 msg="Tracing will be disabled"
level=info ts=2021-01-12T22:00:53.759997096Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2021-01-12T22:00:53.760672294Z caller=inmemory.go:172 msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
level=info ts=2021-01-12T22:00:53.761155984Z caller=options.go:23 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2021-01-12T22:00:53.76178291Z caller=store.go:394 msg="starting store node"
level=info ts=2021-01-12T22:00:53.761881697Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2021-01-12T22:00:53.761842366Z caller=store.go:326 msg="initializing bucket store"
level=info ts=2021-01-12T22:00:53.761908496Z caller=http.go:58 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902

I now get stuck at the same step as you. Can your thanos-store node actually reach your minio/s3 endpoint? However after
X seconds i do get a timeout in thanos-store and it does shutdown...

level=warn ts=2021-01-12T22:07:01.441557567Z caller=intrumentation.go:54 msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Get \"http://127.0.0.1:9000/thanos/?location=\": net/http: timeout awaiting response headers"
level=info ts=2021-01-12T22:07:01.443063232Z caller=http.go:65 service=http/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Get \"http://127.0.0.1:9000/thanos/?location=\": net/http: timeout awaiting response headers"