TLS certificate/key file are not validated

[Zophar78]

Thanos, Prometheus and Golang version used:
thanos, version 0.23.2 (branch: HEAD, revision: fb97c9a)
build user: root@18055e4b6448
build date: 20211222-11:40:52
go version: go1.16.12
platform: linux/amd64

Object Storage Provider:
N/A

What happened:
The certification file/key provided (--grpc-server-tls-cert and --grpc-server-tls-key) are not validated/checked by thanos. I started a thanos sidecar providing those two parameters and my files were not accessible (because of permissions issues). I pushed the tests forward and even provided some non existing files. Each time the Thanos sidecar started normally in healthy state. But at the end i had to deal with TLS issues (Normal) without any clues on the real reasons.
Mar 9 13:23:24 cs1cloudmon1a thanos[62706]: level=info ts=2022-03-09T12:23:24.242283712Z caller=options.go:31 protocol=gRPC msg="enabling server side TLS" Mar 9 13:23:24 cs1cloudmon1a thanos[62706]: level=info ts=2022-03-09T12:23:24.250804965Z caller=sidecar.go:326 msg="starting sidecar" Mar 9 13:23:24 cs1cloudmon1a thanos[62706]: level=info ts=2022-03-09T12:23:24.251328273Z caller=intrumentation.go:60 msg="changing probe status" status=healthy

On client side (like thanos query) for sure i get the following log:
Mar 9 13:44:58 cs1grafana1a thanos[31680]: level=warn ts=2022-03-09T12:44:58.548021448Z caller=endpointset.go:525 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from cs1cloudmon1a.hrp.mydomain.net:10901: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: remote error: tls: internal error\"" address=cs1cloudmon1a.mydomain.net:10901

What you expected to happen:
Thanos should verify and validate the certificate/key provided (At least these file should be readable and reachable from the thanos process) and return a sepcific log message about this non accessible file.

How to reproduce it (as minimally and precisely as possible):
Execute a thanos sidecar with --grpc-server-tls-cert and --grpc-server-tls-key that point to unexisting files

Anything else we need to know:
I don't know if it's by design but it will save some debug/incident resolve time in the future ;)

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.