client and server quick start guides

client_quick_setup.md

@@ -0,0 +1,94 @@
+# Quick Start Guide to use Quobyte from Kubernetes
+
+## Prerequisites
+If you use Docker 1.12 or older, you need to  add the MountFlags parameter.
+For newer versions of Docker, this is not required anymore.
+
+Ensure MountFlags are shared or not set:
+```bash
+$ systemctl cat docker.service | grep MountFlags=shared
+```
+
+If not you can set the MountFlags as shared with the following commands:
+```bash
+$ cat << EOF > /etc/systemd/system/docker.service.d/slave-mount-flags.conf
+[Service]
+MountFlags=shared
+EOF
+
+$ systemctl daemon-reload && systemctl restart docker
+```
+
+## Client setup
+This guide assumes that you have a dedicated Quobyte instance running and you
+want to provide access to Quobyte volumes to pods running in Kubernetes.
+
+To access a Quobyte volume a pod has to run on a Kubernetes node which has a
+Quobyte client running. The client runs inside of a Pod and makes the Quobyte
+storage accessible to other pods.
+
+Quobyte clients run in the quobyte namespace.
+```bash
+$ cd deploy
+$ kubectl create -f quobyte-ns.yaml
+```
+
+To connect to Quobyte, the client needs to resolve the address of the registry.
+It is configured in the client-ds.yaml DaemonSet definition:
+```yaml
+env:
+  - name: QUOBYTE_REGISTRY
+    value: registry.quobyte
+```
+
+If you have a certificate for the client, it is stored as a Secret and
+mounted into the client Pod as client.cfg.
+
+First create a file that contains only the certificate information
+(<ca>, <cert>, and <key> blocks) and store it as a secret.
+```bash
+kubectl -n quobyte create secret generic client-config --from-file /tmp/client.cfg
+```
+
+```bash
+$ kubectl -n quobyte create -f client-ds.yaml
+or
+$ kubectl -n quobyte create -f client-certificate-ds.yaml
+```
+
+The deployed DaemonSet starts client pods on all nodes marked as `quobyte_client`.
+This can either be done manually, or by using the Quobyte operator.
+
+```bash
+$ kubectl label nodes <node-1> <node-n> quobyte_client="true"
+```
+
+When the client pod is up and running, you should see a mount point on the Kubernetes node
+at `/var/lib/kubelet/plugins/kubernetes.io~quobyte`.
+
+##Benchmarking
+For easy testing and benchmarking we provide a fio-container which uses
+Quobyte volumes. By default, it will start writing to volume `fio-test`
+
+```bash
+$ kubectl create -f fio-benchmark-ds.yaml
+```
+This will start a single Pod on a node which is marked as quobyte_client.
+The container is designed to put load on the volume, so you can scale it:
+
+```bash
+kubectl scale --replicas=100 deployment fio-benchmark
+```
+
+## TODO
+volumes
+
+persisted volume claims
+
+storage classes
+
+[1] https://github.com/kubernetes/examples/tree/master/staging/volumes/quobyte
+
+[2] https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#create-a-persistentvolume
+
+[3] https://kubernetes.io/docs/concepts/storage/storage-classes/#quobyte

deploy/README.md

@@ -18,7 +18,8 @@ Kubernetes will make sure that the services are scheduled on the nodes.
 ## Prerequisites
 ### Format Quobyte Devices
 Quobyte is designed to run on dedicated disk drives which are formatted with ext4 or xfs and
-are initialized as a Quobyte device.
+are initialized as a Quobyte device. You can either format and prepare devices manually,
+or let the Quobyte Device Inspector help.
 
 Log in the host machines and prepare the devices. In this example, we expect
 `/dev/sd{b,c,d}` to be available.

deploy/client-certificate-ds.yaml

@@ -0,0 +1,124 @@
+apiVersion: apps/v1beta2
+kind: DaemonSet
+metadata:
+  name: client
+  namespace: quobyte
+spec:
+  selector:
+    matchLabels:
+      role: client
+  template:
+    metadata:
+      annotations:
+        prometheus.io/scrape: 'true'
+        prometheus.io/path: '/prometheus'
+        prometheus.io/port: '55000'
+      labels:
+        role: client
+        version: "2.0.pre4"
+    spec:
+      containers:
+      - name: quobyte-client
+        image: quay.io/quobyte/quobyte-client:2.0.pre4
+        imagePullPolicy: Always
+        # NOTE(kaisers): The weird if check before mkdir -p is required to not hang on a stale mount
+        command:
+          - /bin/sh
+          - -xec
+          - |
+            if [[ ! -f /etcfs/fuse.conf ]]; then
+              echo "Copy fuse config to host"
+              { echo -e '# Copied from Quobyte Client Container\n'; cat /etc/fuse.conf; } > /etcfs/fuse.conf
+            fi
+            if [[ $(grep "^[^#]" /etcfs/fuse.conf | grep -c "user_allow_other") -eq 0 ]]; then
+              echo "user_allow_other" >> /etcfs/fuse.conf
+            fi
+            if cut -d" " -f2 /etcfs/mtab | grep -q ${QUOBYTE_MOUNT_POINT}; then
+              umount ${QUOBYTE_MOUNT_POINT}
+            else
+              if ! [[ $(ls `dirname ${QUOBYTE_MOUNT_POINT}`|egrep "^`basename ${QUOBYTE_MOUNT_POINT}`$") ]]; then
+                echo "mount point ${QUOBYTE_MOUNT_POINT} does not exist, creating it..."
+                mkdir -p ${QUOBYTE_MOUNT_POINT}
+              fi
+            fi
+            if echo ${QUOBYTE_REGISTRY} | grep -q ","; then
+              # registries are outside the k8s cluster, pass through and DNS lookup later
+              ADDR=${QUOBYTE_REGISTRY}
+            else
+              # Currently, within the nsenter, k8s dns names cannot be resolved.
+              ADDR=$(echo $(nslookup ${QUOBYTE_REGISTRY} | grep -A10 -m1 -e 'Name:' | grep Address | awk '{split($0,a,":"); print a[2]}'  | awk '{print $1":7861"}') | tr ' ' ,)
+            fi
+            echo "QUOBYTE_REGISTRY: ${ADDR}"
+            mkdir -p /root/.quobyte
+            cp /quobytecert/client.cfg /root/.quobyte/client.cfg
+            /bin/nsenter -t 1 --wd=. -m -- \
+              lib/ld-linux-x86-64.so.2 \
+              --library-path ./lib \
+            ./bin/mount.quobyte \
+              -c ./root/.quobyte/client.cfg \
+              --hostname ${NODENAME} \
+              --allow-usermapping-in-volumename \
+              --http-port 55000 \
+              -f \
+              -l /dev/stdout \
+              -d ${QUOBYTE_CLIENT_LOG_LEVEL} \
+              ${OPTS} \
+              ${ADDR}/ ${QUOBYTE_MOUNT_POINT}
+        securityContext:
+          privileged: true
+        env:
+          - name: QUOBYTE_CLIENT_LOG_LEVEL
+            value: INFO
+          - name: QUOBYTE_REGISTRY
+            value: registry.quobyte
+          - name: QUOBYTE_MOUNT_POINT
+            # Note(kaisers): the mount point has to be a subdir of the volume(Mount)
+            value: /var/lib/kubelet/plugins/kubernetes.io~quobyte
+          - name: NODENAME
+            valueFrom:
+              fieldRef:
+                fieldPath: spec.nodeName
+        ports:
+          - name: http-port
+            containerPort: 55000
+            hostPort: 55000
+            protocol: TCP
+        readinessProbe:
+          timeoutSeconds: 5
+          httpGet:
+            port: 55000
+            path: /
+        livenessProbe:
+          initialDelaySeconds: 30
+          timeoutSeconds: 5
+          httpGet:
+            port: 55000
+            path: /
+        volumeMounts:
+          - name: k8s-plugin-dir
+            mountPath: /var/lib/kubelet/plugins/kubernetes.io~quobyte
+          - name: etcfs
+            mountPath: /etcfs
+          - name: configs
+            mountPath: /quobytecert
+        lifecycle:
+          preStop:
+            exec:
+              command: ["/bin/sh", "-xc", "/bin/nsenter -t 1 --wd=. -m -- lib/ld-linux-x86-64.so.2 --library-path ./lib ./bin/umount -f ${QUOBYTE_MOUNT_POINT}"]
+      hostPID: true
+      nodeSelector:
+        quobyte_client: "true"
+      volumes:
+      - name: k8s-plugin-dir
+        hostPath:
+          path: /var/lib/kubelet/plugins/kubernetes.io~quobyte
+      - name: etcfs
+        hostPath:
+          path: /etc
+      - name: configs
+        secret:
+          defaultMode: 420
+          items:
+          - key: client.cfg
+            path: client.cfg
+          secretName: client-config

deploy/fio-benchmark-ds.yaml

@@ -37,6 +37,8 @@ spec:
             valueFrom:
               fieldRef:
                 fieldPath: spec.nodeName
+      nodeSelector:
+        quobyte_client: "true"
       volumes:
       - name: quobytevolume
         quobyte:

deploy/registry-bootstrap-ds.yaml

@@ -0,0 +1,115 @@
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: registry
+  namespace: quobyte
+spec:
+  template:
+    metadata:
+      labels:
+        role: registry
+        version: "2.0.pre4"
+    spec:
+      containers:
+      - name: quobyte-registry
+        image: quay.io/quobyte/quobyte-server:2.0.pre4
+        # imagePullPolicy: Always
+        # resources:
+        #   limits:
+        #     memory: "512Mi"
+        #     cpu: "100m"
+
+        # priviliged is required for full device inspector support
+        securityContext:
+          privileged: true
+        env:
+          - name: QUOBYTE_SERVICE
+            value: registry
+          - name: QUOBYTE_REGISTRY
+            value: registry
+          - name: QUOBYTE_EXTRA_SERVICE_CONFIG
+            value: >
+              constants.automation.manage_registry_replicas=true
+          - name: MAX_MEM
+            valueFrom:
+              configMapKeyRef:
+                name: quobyte-config
+                key: registry.max_mem
+          - name: MIN_MEM
+            valueFrom:
+              configMapKeyRef:
+                name: quobyte-config
+                key: registry.min_mem
+        command:
+          - /bin/bash
+          - -xec
+          - |
+            sed "s/.*MIN_MEM_REGISTRY=.*/MIN_MEM_REGISTRY=${MIN_MEM}/" -i /etc/default/quobyte
+            sed "s/.*MAX_MEM_REGISTRY=.*/MAX_MEM_REGISTRY=${MAX_MEM}/" -i /etc/default/quobyte
+            mkdir -p /var/lib/quobyte/devices/registry-bootstrap
+            if [ ! -f /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_SETUP ]; then
+              mkdir -p /devices/dev1
+              cat > /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_SETUP <<EOF
+            device.serial=$(uuidgen)
+            device.model=Kubernetes-hostDir
+            device.type=DIR_DEVICE
+            EOF
+              if [ $(nslookup ${QUOBYTE_REGISTRY} | grep -A10 -m1 -e 'Name:' | wc -l) == 0 ]; then
+                echo "device.bootstrap=true" >> /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_SETUP
+              fi
+            fi
+
+            if [ ! -f /var/lib/quobyte/devices/registry-bootstrap/UUID ]; then
+              echo uuid=$(uuidgen) >> /var/lib/quobyte/devices/registry-bootstrap/UUID
+            fi
+            cat /var/lib/quobyte/devices/registry-bootstrap/UUID >> /etc/quobyte/$QUOBYTE_SERVICE.cfg
+
+            exec /bin/bash -x /opt/main.sh
+        lifecycle:
+          preStop:
+            exec:
+              command:
+                - /bin/bash
+                - -xe
+                - |
+                  qmgmt -u api registry remove $(grep device.id= /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_ID | cut -d= -f2)
+                  rm -rf /var/lib/quobyte/devices/registry-bootstrap/
+        ports:
+          - name: rpc-tcp
+            containerPort: 7861
+            protocol: TCP
+          - name: rpc-udp
+            containerPort: 7861
+            protocol: UDP
+          - name: http
+            containerPort: 7871
+            protocol: TCP
+        volumeMounts:
+          - mountPath: /lib/modules
+            name: libmodules
+          - mountPath: /var/lib/quobyte
+            name: var-lib-quobyte
+            mountPropagation: Bidirectional
+        readinessProbe:
+          timeoutSeconds: 5
+          httpGet:
+            port: 7871
+            path: /
+        livenessProbe:
+          initialDelaySeconds: 30
+          timeoutSeconds: 5
+          httpGet:
+            port: 7871
+            path: /
+      nodeSelector:
+        quobyte_bootstrap: "true"
+      volumes:
+        - name: libmodules
+          hostPath:
+            path: /lib/modules
+        - name: mnt-quobyte
+          hostPath:
+            path: /mnt/quobyte
+        - name: var-lib-quobyte
+          hostPath:
+            path: /var/lib/quobyte

operator/src/operator/Dockerfile.scratch

@@ -1,4 +1,5 @@
 FROM scratch
 ADD operator /
 COPY ./pkg/artifacts /pkg/artifacts
-CMD ["/operator"]
+CMD ["/operator"]
+