Fully automate CI/CD process and introduce manual trigger for release process

[axsaucedo]

This issue contains insight around the CI/CD process and how it's set up for the project. The current objective aims to introduce the following process leveraging Jenkins X for most of the automation steps:

PR Process

Automated process when PR is created (or modified):

• New PR is submitted
• Unit tests for Java, Go and Python are evaluated
• Linting for Java, Go and Python are evaluated
• Run end-to-end tests (maybe - depending on how long)

Manual actions that should also be possible:

• Being able to trigger specific PROW commands (specifically to trigger e2e tests from PR)
  • This would be something like /e2e

Land to master

Automated process when commit is added to master (i.e. landing PR):

• Unit tests for Java, Go and Python are evaluated
• Linting for Java, Go and Python are evaluated
• Run end-to-end tests
• Build all images
• Push (and REPLACE) all images as v0.MAJOR.MINOR-SNAPSHOT
• No TAG / RELEASE should be created (Currently it creates a tag even if the tag is overriden)

Release is not performed automatically, but instead in a separate manual step (below)

Perform release

Seldon team manually creates a tag, which is related to a specific branch - this then triggers the release process, which would be the same as the one above, but it would be bumping a version (and removing the SNAPSHOT suffix).

This could be triggered by running a manual script that would bump the version (i.e. change all the SNAPSHOT) into the actual version, then commit and tag, which then should trigger the release).

Automated process when triggered:

• Unit tests for Java, Go and Python are evaluated
• Linting for Java, Go and Python are evaluated
• Run end-to-end tests
• Build all images
• Push all images as v0.MAJOR.MINOR-RC
• Create changelog based on the commit titles

What is not currently automated (and hence need to be performed manually):

• Security scan (currently can be done with AWS)
• Licenses scan (making sure we don’t commit GPL2)
• Code coverage (could be part of the build)

[axsaucedo]

Currently there was some exploratory work to identify how to run KIND in Kubernetes. It seems that the way to correctly run it (based on kubernetes-sigs/kind#303) it is by running a kubernetes pod with mounted volumes and a docker service running inside of the container.

[axsaucedo]

An example Kubernetes pod that would be able to start the docker service and create a KIND cluster would require mounted volumes as follows:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: k8s-builder
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: k8s-101
    spec:
      containers:
      - name: k8s-docker-builder
        image: seldonio/core-builder:0.4
        imagePullPolicy: Always
        command: 
        - tail 
        args:
        - -f 
        - /dev/null
        volumeMounts:
        - mountPath: /lib/modules
          name: modules
          readOnly: true
        - mountPath: /sys/fs/cgroup
          name: cgroup
        - name: dind-storage
          mountPath: /var/lib/docker
        securityContext:
          privileged: true
      volumes:
      - name: modules
        hostPath:
          path: /lib/modules
          type: Directory
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup
          type: Directory
      - name: dind-storage
        emptyDir: {}

The docker image itself would require installing Docker, KIND and a few other dependencies, which are well outlined Kubernetes Sig-infra KIND Runtime Environment (KRTE). You can see the Dockerfile which provides some of the dependencies required, as well as the wrapper.sh which is the entry point that starts and stops the docker service.

[axsaucedo]

As its outlined in kubernetes-sigs/kind#303 (comment) we need to be careful with stopping the KIND cluster and shutting down the docker service.

[axsaucedo]

Initial milestone to run e2e tests is not added through #994. That current PR includes:
This PR is ready to be reviewed. The PR contains:

• Updates to jenkins-x file to run e2e files every time PR is landed
• Override in jenkins-x that avoids the tagged version being created
• Script to run e2e tests inside docker-enabled pod
• Updated core-builder:0.4 with capability to build Go Operator, Python & Java engine

More specifically, what the PR will do, is that it will run the e2e tests using KIND (kubernetes in docker) inside the a Jekins X triggered pod. The next steps will be the following:

Adding a notification if e2e tests fail:

• Agree what would be a good way to be notified if the e2e tests fail, which could be added to the kind_test_all.sh script (could be as simple as a curl to the Slack API so we get a notification)

Build and push the docker images ONLY to SNAPSHOT for every PR merged

This would involve two things, the first one is modifying the jenkins-x file to have the commands to run this, such as the following:

buildPack: none
pipelineConfig:
  pipelines:
    pullRequest:
      pipeline:
        agent:
          image: seldonio/core-builder:0.3
        stages:
        - name: build-and-test
          parallel:
          - agent:
              image: seldonio/python-builder:0.2
            name: seldon-python
            steps:
            - name: test-python
              args:
              - -C python
              - update_package
              - install
              - test
              command: make
            - name: test-python-tf
              args:
              - -C python
              - update_package
              - install-tf
              - test
              command: make
          - name: seldon-engine
            agent:
              image: seldonio/core-builder:0.4
            steps:
            - name: test-engine
              args:
              - -C engine
              - -f Makefile.ci
              - build_jar
              command: make
          - name: build-and-push-images
            agent:
              image: seldonio/core-builder:0.4
            steps:
            - name: build-and-push-engine
              command: make
              args:
              - -C engine
              - build_image
              - push_to_registry
            - name: build-and-push-python-wrapper
              command: cd
              args:
              - wrappers/s2i/python/build_scripts && 
              - bash
              - ./build_all.sh &&
              - bash
              - ./push_all.sh
            - name: build-and-push-operator
              command: make
              args:
              - -C operator
              - docker-build
              - docker-push
            - name: build-and-push-model-servers
              command: cd
              args:
              - wrappers/s2i/python/build_scripts && 
              - bash
              - ./build_all.sh &&
              - bash
              - ./push_all.sh
    release:
      pipeline:
        agent:
          image: seldonio/core-builder:0.4
        stages:
          - name: end-to-end
            steps:
            - name: test-end-to-end
              command: cd
              args:
              - testing/scripts &&
              - bash
              - kind_test_all.sh
            options:
              containerOptions:
                volumeMounts:
                  - mountPath: /lib/modules
                    name: modules
                    readOnly: true
                  - mountPath: /sys/fs/cgroup
                    name: cgroup
                  - name: dind-storage
                    mountPath: /var/lib/docker
                resources:
                  requests:
                    cpu: 1
                    memory: 4000Mi
                securityContext:
                  privileged: true
                imagePullPolicy: Always
              volumes:
                - name: modules
                  hostPath:
                    path: /lib/modules
                    type: Directory
                - name: cgroup
                  hostPath:
                    path: /sys/fs/cgroup
                    type: Directory
                - name: dind-storage
                  emptyDir: {}

Furthermore, in order to run these commands we will have to use the core-builder:0.4 in a similar way to how we are building the kind tests (i.e. run the docker daemon with the volumes mounted and run the docker build / push commands.