Add limited internal storage and postgresql (#3813)

* add loop storage

* fix

* fix

* fix mountPropagation

* add postgresql

* fix

* fix

* fix node affinity for delete.yaml

* fix enable for service

* fix

* add docs

* Update README.md

* add fault tolerance for internal storage

* minor docs change

* Update README.md

* Update README.md

* link db with storage

* fix doc

* fix doc

* fix delete

* fix path

* fix

* fix: allow multiple master; postgresql port; add rest-server env

* fix: doc

* minor doc fix

* fix: disable

src/internal-storage/README.md

@@ -0,0 +1,92 @@
+## PAI Internal Storage
+
+Internal Storage is designed to create a limited size storage in PAI. The storage can be used by database service or other stateful application internally. It leverages [`loop device`](http://man7.org/linux/man-pages/man4/loop.4.html) in Linux to provide a storage with strictly limited quota. The default service configuration for internal storage is:
+
+```yaml
+internal-storage:
+    enable: false
+    type: hostPath
+    root-path: /mnt/paiInternal
+    quota-gb: 10
+```
+
+User can override these settings in `services-configuration.yaml`.
+
+## Set up Internal Storage
+
+For now, `hostPath` is the only supported `type` for internal storage. In summary, it will make a `<root-path>` folder (The default path is `/mnt/paiInternal`) on the `pai-master` node first, then create a loop device in the folder. If the path does not exist, PAI will create it for you. Please refer to the following commands for details of loop device creation.
+
+```bash
+fallocate -l ${QUOTA_GB}G storage.ext4
+/sbin/mkfs -t ext4 -q storage.ext4 -F
+mkdir -p storage
+mount -o loop,rw,usrquota,grpquota storage.ext4 storage
+```
+
+The advantage of using a loop device is that it can limit the disk quota for every user strictly. 
+
+Since the service uses a `mount` inside a container, `mountPropagation` is set to `Bidirectional` to ensure the `mount` behavior propagates to the host.
+
+
+## Use the Internal Storage
+
+In fact, the internal storage is a disk path on the `pai-master` node, thus only pod on the same node can reference it by using `hostPath` in kubernetes, e.g.
+
+```yaml
+apiVersion: v1
+kind: Pod
+...
+spec:
+  affinity:
+    nodeAffinity:
+      requiredDuringSchedulingIgnoredDuringExecution:
+        nodeSelectorTerms:
+        - matchExpressions:
+          - key: kubernetes.io/hostname
+            operator: In
+            values:
+            - {{ cluster_cfg["internal-storage"]["master-ip"] }}
+  containers:
+  - image: <image-name>
+    volumeMounts:
+    - name: internal-data-dir
+      mountPath: /data
+      mountPropagation: "None"
+  volumes:
+  - name: internal-data-dir
+    hostPath:
+      path: '{{ cluster_cfg["internal-storage"]["root-path"] }}/storage'
+```
+
+Please note that `mountPropagation` should be set to `None`, to ensure that any unexpected unmount of the data folder will not be propagates to the pod.
+
+## Assumption of Failure
+
+### 1. Failure during setup
+
+This service uses the readiness probe in k8s to ensure the corresponding loop device is created successfully. Possible errors during setup are as follows:
+
+  - Allocation Failure: The storage uses `fallocate` to reserve quota during setup. If the remaining disk size doesn't meet the need, allocation failure happens.
+  - Mount Failure: Since the `mount` command needs some privileges from the host to work, it may also fail during setup.
+
+If any of the above failures happens, the service will never be ready (because of the readiness probe). See [create.sh](src/create.sh) and [create.yaml.template](deploy/create.yaml.template) for details.
+
+### 2. Failure after setup
+
+Please note that this storage doesn't have any replica mechanism. If the `pai-master` node crashes with a disk failure or other hardware issues, users will not be able to restore the data. In fact, all the data are stored in a single file `storage.ext4` on the `pai-master` node.
+
+Possibility is that users may delete our storage file `storage.ext4` or `storage` folder unexpectedly. The service checks them every 60 seconds:
+
+  - If the `storage` folder is unmounted or deleted, the service will restart to create and mount it again in 60 seconds. Data won't be lost. Since pods are using the internal storage with `mountPropagation=None`, they won't notice any change.
+  - If the `storage.ext4` file is deleted, the service will restart to create a new `storage.ext4` in 60 seconds. However, in such case, user data will be lost. We cannot prevent it since users can always remove files on their disks.
+
+
+### 3. Failure during deletion 
+
+During service deletion, if we cannot unmount or delete the data, the deletion process won't be successful. There is also a readiness probe for these purposes. See [delete.yaml.template](deploy/delete.yaml.template) for details.
+
+
+## References
+  - [Loop Device](http://man7.org/linux/man-pages/man4/loop.4.html)
+  - [Linux Quota Tutorial](http://souptonuts.sourceforge.net/quota_tutorial.html)
+  - [Mount Propagation](https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation)

src/internal-storage/build/internal-storage-create.dockerfile

@@ -0,0 +1,24 @@
+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+FROM ubuntu:16.04
+
+RUN mkdir -p /init_scripts
+
+COPY src/create.sh /init_scripts
+
+ENTRYPOINT /bin/bash /init_scripts/create.sh

src/internal-storage/build/internal-storage-delete.dockerfile

@@ -0,0 +1,24 @@
+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+FROM ubuntu:16.04
+
+RUN mkdir -p /init_scripts
+
+COPY src/delete.sh /init_scripts
+
+ENTRYPOINT /bin/bash /init_scripts/delete.sh

src/internal-storage/config/internal-storage.yaml

@@ -0,0 +1,4 @@
+enable: false
+type: hostPath
+root-path: /mnt/paiInternal
+quota-gb: 10

src/internal-storage/config/internal_storage.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python
+import copy
+import logging
+
+
+class InternalStorage(object):
+    def __init__(self, cluster_conf, service_conf, default_service_conf):
+        self.cluster_conf = cluster_conf
+        self.service_conf = self.merge_service_configuration(service_conf, default_service_conf)
+        self.logger = logging.getLogger(__name__)
+
+    @staticmethod
+    def merge_service_configuration(overwrite_srv_cfg, default_srv_cfg):
+        if overwrite_srv_cfg is None:
+            return default_srv_cfg
+        srv_cfg = default_srv_cfg.copy()
+        for k in overwrite_srv_cfg:
+            srv_cfg[k] = overwrite_srv_cfg[k]
+        return srv_cfg
+
+    def validation_pre(self):
+        if self.service_conf['enable']:
+            type_ = self.service_conf.get('type', '')
+            if type_ == 'hostPath':
+                machine_list = self.cluster_conf['machine-list']
+                if len([host for host in machine_list if host.get('pai-master') == 'true']) < 1:
+                    return False, '"pai-master=true" machine is required to deploy the internal storage'
+                quotaGB = int(self.service_conf['quota-gb'])
+                assert quotaGB >= 1
+                return True, None
+            else:
+                return False, 'Unknown internal storage type {}'.format(type_)
+        else:
+            return True, None
+
+    def run(self):
+        result = copy.deepcopy(self.service_conf)
+        if result['enable']:
+            machine_list = self.cluster_conf['machine-list']
+            master_ip = [host['hostip'] for host in machine_list if host.get('pai-master') == 'true'][0]
+            result['master-ip'] = master_ip
+            result['quota-gb'] = int(result['quota-gb'])
+        return result
+
+    def validation_post(self, conf):
+        return True, None

src/internal-storage/deploy/create.yaml.template

@@ -0,0 +1,68 @@
+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: internal-storage-create-ds
+spec:
+  selector:
+    matchLabels:
+      app: internal-storage-create
+  template:
+    metadata:
+      name: internal-storage-create
+      labels:
+        app: internal-storage-create
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+            - matchExpressions:
+              - key: kubernetes.io/hostname
+                operator: In
+                values:
+                - {{ cluster_cfg["internal-storage"]["master-ip"] }}
+      hostNetwork: false
+      containers:
+      - name: internal-storage-create
+        image: {{ cluster_cfg["cluster"]["docker-registry"]["prefix"] }}internal-storage-create:{{ cluster_cfg["cluster"]["docker-registry"]["tag"] }}
+        securityContext:
+          privileged: true
+        imagePullPolicy: Always
+        readinessProbe:
+          exec:
+            command:
+            - ls
+            - /paiInternal/storage/READY
+          initialDelaySeconds: 10
+          periodSeconds: 3
+        env:
+        - name: QUOTA_GB
+          value: '{{ cluster_cfg["internal-storage"]["quota-gb"] }}'
+        volumeMounts:
+        - name: internal-data-dir
+          mountPath: /paiInternal
+          mountPropagation: Bidirectional
+      volumes:
+        - name: internal-data-dir
+          hostPath:
+            path: {{ cluster_cfg["internal-storage"]["root-path"] }}
+            type: DirectoryOrCreate
+      imagePullSecrets:
+      - name: {{ cluster_cfg["cluster"]["docker-registry"]["secret-name"] }}

src/internal-storage/deploy/delete.sh.template

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+pushd $(dirname "$0") > /dev/null
+
+kubectl delete --ignore-not-found --now "daemonset/internal-storage-create-ds"
+
+{% if cluster_cfg['internal-storage']['enable'] %}
+
+kubectl apply --overwrite=true -f delete.yaml || exit $?
+
+# Wait until the service is ready.
+PYTHONPATH="../../../deployment" python -m  k8sPaiLibrary.monitorTool.check_pod_ready_status -w -k app -v internal-storage-delete || exit $?
+
+kubectl delete --ignore-not-found --now "daemonset/internal-storage-delete-ds"
+
+{% endif %}
+
+popd > /dev/null

src/internal-storage/deploy/delete.yaml.template

@@ -0,0 +1,64 @@
+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: internal-storage-delete-ds
+spec:
+  selector:
+    matchLabels:
+      app: internal-storage-delete
+  template:
+    metadata:
+      name: internal-storage-delete
+      labels:
+        app: internal-storage-delete
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+            - matchExpressions:
+              - key: kubernetes.io/hostname
+                operator: In
+                values:
+                - {{ cluster_cfg["internal-storage"]["master-ip"] }}
+      hostNetwork: false
+      containers:
+      - name: internal-storage-delete
+        image: {{ cluster_cfg["cluster"]["docker-registry"]["prefix"] }}internal-storage-delete:{{ cluster_cfg["cluster"]["docker-registry"]["tag"] }}
+        securityContext:
+          privileged: true
+        imagePullPolicy: Always
+        readinessProbe:
+          exec:
+            command:
+            - ls
+            - /DELETED
+          initialDelaySeconds: 10
+          periodSeconds: 3
+        volumeMounts:
+        - name: internal-data-dir
+          mountPath: /paiInternal
+          mountPropagation: Bidirectional
+      volumes:
+        - name: internal-data-dir
+          hostPath:
+            path: {{ cluster_cfg["internal-storage"]["root-path"] }}
+      imagePullSecrets:
+      - name: {{ cluster_cfg["cluster"]["docker-registry"]["secret-name"] }}

src/internal-storage/deploy/refresh.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+pushd $(dirname "$0") > /dev/null
+
+bash stop.sh
+bash start.sh
+
+popd > /dev/null