Polyaxon deployment on local kubernetes cluster (postgresql persistence)

[fzoric8]

I've been trying to setup Polyaxon on local cluster with kubernetes for over a week. I can't get a handle on postgresql setup, therefore I'm not able to finish whole process. I'm trying to create persistentVolumeClaim which I'll use as existingClaim for postgresql data persistence. PersistentVolume and PersistentVolumeClaim are created successfuly, and every other pod in cluster is running except postgresql pod!
I've been stuck with same error message that shows up during postgresql pod creation and results in ContainerConfigError.

ERROR MESSAGE:

Error: failed to create subPath directory for volumeMount "data" of container "polyaxon-postgresql"

I'm using nfs for creating PV and PVCs, and folders used are exposed in /etc/exports properly.

I've doubt it's bug, it looks like some kind of permission error, but I haven't found anything related. Help would be really appreciated.

I've provided tutorials used, current versions of docker, kubernetes, helm and config files for creating PersistentVolume, PersistentVolumeClaim and polyaxon_config.yaml.

Mainly I've been following these instructions:

• kubedam
• kubernetes
• postgresql-ha

Environment:

linux version:

NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

kubectl version:

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"7a578febe155a7366767abce40d8a16795a96371", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

kubeadm version:

kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:50:16Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

helm version:

Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

docker version:

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

posgresql-pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: polyaxon-pv-postgresql
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: <master_IP>
    path: "/k8s-storage/storage/postgresql"
  claimRef:
    namespace: polyaxon
    name: polyaxon-pvc-postgresql
  persistentVolumeReclaimPolicy: Retain
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: polyaxon-pvc-postgresql
  namespace: polyaxon
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

polyaxon_config.yaml

# Role-based access control:
rbac:
  enabled: true

# Online availability (see Kubernetes service types for more info):
ingress:
  enabled: false

serviceType: NodePort

# First user (it's also admin):
user:
  username: "root"
  ...

postgresql:
  persistence:
    enabled: true
    existingClaim: polyaxon-pvc-posgresql

# Persistence of data:
persistence:
  logs:
    existingClaim: polyaxon-pvc-logs
  repos:
    existingClaim: polyaxon-pvc-repos
  upload:
    existingClaim: polyaxon-pvc-upload

  data:
      data:
          existingClaim: polyaxon-pvc-data
          mountPath: /data
  outputs:
      output:
          existingClaim: polyaxon-pvc-outputs
          mountPath: /outputs
redis:
  persistence:
    existingClaim: polyaxon-pvc-redis-master
  master:
    nodeSelector:
      polyaxon: kubernetes-master
    tolerations:
      - key: "node-role.kubernetes.io/master"
        effect: "NoSchedule"
        operator: "Exists"
  slave:
    nodeSelector:
      polyaxon: kubernetes-master
    tolerations:
      - key: "node-role.kubernetes.io/master"
        effect: "NoSchedule"
        operator: "Exists"

[fzoric8]

I've managed to solve this problem.

Firstly I added subPath argument to polyaxon_config.yml such as:

postgresql:
  persistence: true
  existingClaim: <pvc_name>
  subPath: ""

which resulted in following ERROR while building postgresql container:

chown: changing ownership of '/var/lib/postgresql/data/pgdata/data': Operation not permitted
chown: changing ownership of '/var/lib/postgresql/data/pgdata': Operation not permitted

After that, I was quite sure I've managed to mess something up and it's related with following docker issue when mounting something on nfs.

I've reconfigured postgresql-pvc.yaml as following:

server: <master_ip>
nfs: "/home/postgresql/data"

And polyaxon_config.yaml without subPath declaration related to postgresql:

postgresql:
  persistence:
    enabled: true
    existingClaim: polyaxon-pvc-postgresql

And added following in /etc/exports:

/home/postgresql *(rw,sync,no_subtree_check,no_root_squash)

After that it's necessary to restart nfs with sudo /etc/init.d/nfs-kernel-server restart or sudo exportfs -arv.

Make sure that you have existing /home/postgresql/data directory. That should be it.