Local installation of Sunbird Knowlg using kubernetes

KNOWLG-SETUP.md

@@ -0,0 +1,125 @@
+
+Below are the steps to set up the Sunbird Knowlg Microservices, DBs with seed data and Jobs. It uses a local Kubernetes cluster deploy the required services.
+
+### Prerequisites:
+* Java 11
+* Maven
+* Docker
+* Minikube - It implements a local Kubernetes cluster on macOS, Linux, and Windows.
+* KubeCtl - The Kubernetes command-line tool
+
+### Prepare folders for database data and logs
+
+```shell
+mkdir -p ~/sunbird-dbs/neo4j ~/sunbird-dbs/cassandra ~/sunbird-dbs/redis ~/sunbird-dbs/es ~/sunbird-dbs/kafka
+export sunbird_dbs_path=~/sunbird-dbs
+```
+
+
+
+### Docker Images of Knowlg MicroServices
+Start Docker in your machine and create the Docker Images of below microservices using the shell script.
+1. taxonomy-service
+2. content-service
+3. search-service
+
+```shell
+sh ./knowlg-docker-image.sh <TAG> # provide the TAG for the docker image.
+```
+**Note:** Please specify the TAG for the Docker Images and update the configuration in helm chart of respective deployment.
+
+Check the Docker Images
+```shell
+docker image ls -a
+```
+**Output:**
+```shell
+❯❯ docker image ls -a
+REPOSITORY                    TAG       IMAGE ID       CREATED          SIZE
+assessment-service            R5.0.0    72a9cc1b2cc4   14 seconds ago   479MB
+search-service                R5.0.0    24b7d8947a4f   23 seconds ago   465MB
+content-service               R5.0.0    afcbc9c10fa3   33 seconds ago   556MB
+taxonomy-service              R5.0.0    a8a24a6241f2   47 seconds ago   480MB
+```
+
+### Kubernetes Cluster Setup
+Please use the minikube to quickly set up the kubernetes cluster in local machine.
+
+```shell
+minikube start
+```
+
+### Load Docker Images to Minikube Cluster
+```shell
+minikube image load neo4j:3.3.0
+minikube image load taxonomy-service:R5.0.0
+```
+
+### Create Namespace 
+Create the namespaces to deploy the API microservices, DBs and Jobs.
+1. knowlg-api
+2. knowlg-db
+3. knowlg-job
+
+```shell
+kubectl create namespace knowlg-api
+kubectl create namespace knowlg-db
+kubectl create namespace knowlg-job
+```
+
+### Setup Databases
+Please run the below `helm` commands to set up the required databases within the kubernets cluster.
+It requires the below DBs for Knowlg.
+1. Neo4J
+2. Cassandra
+3. Elasticsearch
+4. Kafka
+5. Redis
+
+```shell
+cd kubernetes
+helm install redis sunbird-dbs/redis -n knowlg-db
+
+minikube mount <LOCAL_SOURCE_DIR>:/var/lib/neo4j/data // LOCAL_SOURCE_DIR is where neo4j dump is extracted Ex: /Users/abc/sunbird-dbs/neo4j/data
+helm install neo4j sunbird-dbs/neo4j -n knowlg-db
+
+minikube mount <LOCAL_SOURCE_DIR>:/mnt/backups // LOCAL_SOURCE_DIR is where neo4j dump is extracted Ex: /Users/abc/sunbird-dbs/cassandra/backups
+helm install cassandra sunbird-dbs/cassandra -n knowlg-db
+
+ssh to cassandra pod
+run => cqlsh
+run => source '/mnt/backups/cassandra_backup/db_schema.cql';
+```
+
+**Note:** 
+- The `helm` charts for Kafka, Elasticsearch will be added soon.
+
+### Define ConfigMap
+We use the configmap to load the configuration for the microservices.
+
+#### ConfigMap for Taxonomy-Service
+Use the below commands to load the configmap of taxonomy-service.
+1. `taxonomy-config` - this has the application configuration. Please update the variables with respect to your context and load.
+2. `taxonomy-xml-config` - this has the logback configuration to handle the logs.
+
+We have to update the below configurations in `taxonomy/templates/taxonomy-service_application.conf` specific to your context.
+
+```shell
+cd kubernetes
+kubectl create configmap taxonomy-xml-config --from-file=taxonomy/taxonomy-service_logback.xml -n knowlg-api -o=yaml
+kubectl create configmap taxonomy-config --from-file=taxonomy/taxonomy-service_application.conf -n knowlg-api  -o=yaml
+```
+
+### Run Taxonomy-Service
+Use the `taxonomy` helm chart to run the taxonomy-service in local kubernetes cluster.
+
+```shell
+cd kubernetes
+helm install taxonomy taxonomy -n knowlg-api
+```
+Use Port Forwarding to access the application in the cluster from local.
+
+```shell
+kubectl port-forward <pod-name> 9000:9000 -n knowlg-api
+curl 'localhost:9000/health'
+```

knowlg-docker-image.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+TAG=$1
+docker rmi $(docker images -a | grep taxonomy-service | awk '{print $1":"$2}')
+docker rmi $(docker images -a | grep content-service | awk '{print $1":"$2}')
+docker rmi $(docker images -a | grep search-service | awk '{print $1":"$2}')
+
+# Taxonomy Service
+cd taxonomy-api/taxonomy-service
+mvn play2:dist
+cd ../..
+docker build -f build/taxonomy-service/Dockerfile  -t taxonomy-service:${TAG} .
+
+# Content Service
+cd content-api/content-service
+mvn play2:dist
+cd ../..
+docker build -f build/content-service/Dockerfile -t content-service:${TAG} .
+
+# Search Service
+cd search-api/search-service
+mvn play2:dist
+cd ../..
+docker build -f build/search-service/Dockerfile -t search-service:${TAG} .

kubernetes/sunbird-dbs/cassandra/Chart.yaml

@@ -0,0 +1,5 @@
+apiVersion: v1
+appVersion: "1.0"
+description: A Helm chart for Kubernetes
+name: cassandra
+version: 0.1.0

kubernetes/sunbird-dbs/cassandra/templates/deployment.yaml

@@ -0,0 +1,33 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ .Chart.Name }}
+  namespace: {{ .Values.namespace }}
+spec: 
+  replicas: {{ .Values.replicaCount }}
+  selector:
+    matchLabels:
+      app: {{ .Chart.Name }}
+  template:
+    metadata:
+      labels:
+        app: {{ .Chart.Name }}
+    spec:
+      containers:
+        - name: {{ .Chart.Name }}
+          image: cassandra:3.11.11
+          ports:
+            - containerPort: {{ .Values.network.port }}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ .Chart.Name }}-db
+  namespace: {{ .Values.namespace }}
+  labels:
+    app: {{ .Chart.Name }}
+spec:
+  ports:
+    - port: {{ .Values.network.port }}
+  selector:
+    app: {{ .Chart.Name }}

kubernetes/sunbird-dbs/cassandra/values.yaml

@@ -0,0 +1,4 @@
+namespace: knowlg-db
+replicaCount: 1
+network:
+  port: 9042

kubernetes/sunbird-dbs/neo4j/Chart.yaml

@@ -0,0 +1,5 @@
+apiVersion: v1
+appVersion: "1.0"
+description: A Helm chart for Kubernetes
+name: neo4j
+version: 0.1.0

kubernetes/sunbird-dbs/neo4j/templates/deployment.yaml

@@ -0,0 +1,51 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ .Chart.Name }}
+  namespace: {{ .Values.namespace }}
+spec: 
+  replicas: {{ .Values.replicaCount }}
+  selector:
+    matchLabels:
+      app: {{ .Chart.Name }}
+  template:
+    metadata:
+      labels:
+        app: {{ .Chart.Name }}
+    spec:
+      containers:
+        - name: {{ .Chart.Name }}
+          image: {{ .Values.image }}
+          imagePullPolicy: Never
+          env: ## Env variables to pass
+            - name: NEO4J_dbms_security_auth__enabled
+              value: "false"
+          ports:
+            - containerPort: 7474
+              name: cypher-port
+            - containerPort: 7687
+              name: bolt-port-1
+            - containerPort: 8687
+              name: bolt-port-2
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ .Chart.Name }}-db
+  namespace: {{ .Values.namespace }}
+  labels:
+    app: {{ .Chart.Name }}
+spec:
+  ports:
+    - name: cypher-shell-port
+      port: 7474
+      targetPort: 7474
+    - name: bolt-port-1
+      port: 7687
+      targetPort: 7687
+    - name: bolt-port-2
+      port: 8687
+      targetPort: 8687
+  selector:
+    app: {{ .Chart.Name }}

kubernetes/sunbird-dbs/neo4j/values.yaml

@@ -0,0 +1,18 @@
+namespace: knowlg-db
+replicaCount: 1
+image: neo4j:3.3.0
+authEnabled: false
+network:
+  port: 7474
+neo4j:
+  name: neo4j
+  resources:
+    cpu: "0.5"
+    memory: "2Gi"
+
+volumes:
+  data:
+    mode: defaultStorageClass
+    defaultStorageClass:
+      requests:
+        storage: 2Gi

kubernetes/sunbird-dbs/redis/Chart.yaml

@@ -0,0 +1,5 @@
+apiVersion: v1
+appVersion: "1.0"
+description: A Helm chart for Kubernetes
+name: redis
+version: 0.1.0

kubernetes/sunbird-dbs/redis/templates/deployment.yaml

@@ -0,0 +1,34 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ .Chart.Name }}
+  namespace: {{ .Values.namespace }}
+spec: 
+  replicas: {{ .Values.replicaCount }}
+  selector:
+    matchLabels:
+      app: {{ .Chart.Name }}
+  template:
+    metadata:
+      labels:
+        app: {{ .Chart.Name }}
+    spec:
+      containers:
+        - name: {{ .Chart.Name }}
+          image: redis
+          ports:
+            - containerPort: {{ .Values.network.port }}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ .Chart.Name }}-db
+  namespace: {{ .Values.namespace }}
+  labels:
+    app: {{ .Chart.Name }}
+spec:
+  ports:
+    - port: {{ .Values.network.port }}
+      targetPort: {{ .Values.network.targetport }}
+  selector:
+    app: {{ .Chart.Name }}

kubernetes/sunbird-dbs/redis/values.yaml

@@ -0,0 +1,5 @@
+namespace: knowlg-db
+replicaCount: 1
+network:
+  port: 6379
+  targetport: 6379

kubernetes/taxonomy/Chart.yaml

@@ -0,0 +1,5 @@
+apiVersion: v1
+appVersion: "1.0"
+description: A Helm chart for Kubernetes
+name: taxonomy
+version: 0.1.0