store recommends setting proper mlock limits which is not possible in upstream docker images

[chrischdi]

Thanos, Prometheus and Golang version used:

thanos, version 0.13.0-rc.2 (branch: release-0.13-caas, revision: 726708a7b05b0c4037ac064c4bb789c4d687cd3d)
  build user:       [email protected]
  build date:       20200616-05:00:59
  go version:       go1.14

Built from https://github.com/c445/thanos/tree/726708a7b05b0c4037ac064c4bb789c4d687cd3d
(this branch is equivalent to the tag v0.13.0-rc.2, except cherry-picking #2752 )

Object Storage Provider:
Cloudian S3

What happened:

On startup thanos-store did panic witht he following error:

level=info ts=2020-06-17T06:07:28.538999812Z caller=store.go:317 msg="initializing bucket store"
runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed

runtime stack:

I tried to increase the mlock limit by executing ulimit -l 512 before executing thanos.
Because /bin/busybox inside the root image (which is based on quay.io/prometheus/busybox:latest lacks additional capabilities.

What you expected to happen:

ulimit -l 512 works.

How to reproduce it (as minimally and precisely as possible):

Run the thanos image as non-root in kubernetes using the following spec:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: test
  serviceName: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - command:
        - sh
        - -c
        - |
          set -ex
          tail -f /dev/null
        image: v0.13.0-rc.2	
        name: test
        securityContext:
          capabilities:
            add:
              - "SYS_RESOURCE"
              - "IPC_LOCK"
      securityContext:
        fsGroup: 65534
        runAsUser: 65534

kubectl exec into the pod and run ulimit -l 512:

/ $ ulimit -l 512
sh: error setting limit: Operation not permitted

Full logs to relevant components:

Anything else we need to know:

I was able to workaround it by rebuilding the docker image using following commit:
c445@bcb0969

TLDR:

• it uses alpine instead of busybox as base image
• it installs libcap and adds capabilities to /bin/busybox

Environment:

• OS (e.g. from /etc/os-release): ubuntu 19.10
• Kernel (e.g. uname -a): 5.3.0-46-generic
• Others:

# from `kubectl get node`
VERSION       OS-IMAGE       KERNEL-VERSION     CONTAINER-RUNTIME
v1.17.6       Ubuntu 19.10   5.3.0-46-generic   containerd://1.3.4

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[chrischdi]

/reopen
/remind

[chrischdi]

This still is an issue to us on the latest release v0.17. We currently have to rebuild the image on our own :-(