Skip to content

Latest commit

 

History

History
178 lines (137 loc) · 5.74 KB

k8s-setup-guide.md

File metadata and controls

178 lines (137 loc) · 5.74 KB

K8S cluster environment setup guide

Welcome to see the guide about the K8S cluster environment.

Preparation

You must have at least three computers,a master-node server,one or more work-node servers and an NFS server.It is recommended to install Ubuntu 20.* or above.

System initialization

  • In order to install kubernetes, you need to install some dependencies on the system and modify some system settings, Please perform step 1 to step 4 on each kubernetes node.

    Step 1 Install dependencies

    $ apt update
$ apt install -y wget conntrack net-tools ntpdate ntp ipvsadm ipset vim jq iptables sysstat

    Step 2 Disable swapoff

    # Temporarily close swapoff
$ swapoff -a 

# Permanently close swapoff
$ sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

    Step 3 Set kernel parameters for kubernetes

    $ vim /etc/sysctl.d/kubernetes.conf

vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv6.conf.all.disable_ipv6=1

$ sysctl -p /etc/sysctl.d/kubernetes.conf

    Step 4 Preparation of ipvs

    $ modprobe br_netfilter
$ vim /etc/systemd/network/ipvs.modules

modprobe -- ip_vs 
modprobe -- ip_vs_rr 
modprobe -- ip_vs_wrr 
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

$ chmod 755 /etc/systemd/network/ipvs.modules && bash /etc/systemd/network/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

Kubernetes installation

  • After the kubernetes node server initialization is completed, we start to install kubernetes. First, you need to execute step 1 to step 3 on each kubernetes node server to complete the preparations for kubernetes initialization, then execute step 4 on the master-node server to initialize, and execute step 5 on each work-node to join the work-node to the master-server. So far, the kubernetes has been installed, If you want to support SGX Device Plugin for Kubernetes, you can perform step 6.

    Step 1 Docker installation

    $ curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
$ mkdir /etc/docker
$ vim /etc/docker/daemon.json

{
    "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn/"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m"
    }
}

$ mkdir -p /etc/systemd/system/docker.service.d

$ systemctl daemon-reload && systemctl restart docker && systemctl enable docker

    Step 2 kubeadm installation

    $ apt-get update && apt-get install -y apt-transport-https

$ curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 

$ cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

$ apt-get update

$ apt-get -y install kubeadm=1.23.5-00 kubectl=1.23.5-00 kubelet=1.23.5-00

$ systemctl enable kubelet.service

    Step 3 Import required Kubernetes cluster images for installation

    Download kubeadm-basic.images.tar.gz and docker load. You can use the following command to view the image you need to import.

    $ kubeadm config images list --kubernetes-version v1.23.5

    Step 4 Initialize master-node

    # Create an initialization yaml file
$ kubeadm config print init-defaults > kubeadm-config.yaml

# Edit yaml file and modify the following contents
$ vim kubeadm-config.yaml

# Change 1.2.3.4 to your master-node's IP
localAPIEndpoint:
  advertiseAddress: 1.2.3.4

# Change node to your master-node's hostname
nodeRegistration:
  name: node


# Check the version is your kubeadm version.
kubernetesVersion: v1.23.5

# Add podSubnet settings.
networking:
dnsDomain: cluster.local
# The 10.244.0.0/16 is Flannel default network segment.
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12

# Add setting for change proxy mode to ipvs.
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs

# Start initialize and save log to kubeadm-init.log
$ kubeadm init --config=kubeadm-config.yaml | tee kubeadm-init.log

# You can find the following commands in kubeadm-init.log and execute them.
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

# flannel installation
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

    Step 5 Join work-node to master-node

    You can find the [kubeadm join] command in kubeadm-init.log and execute it on the work-node.

    $ kubeadm join 1.2.3.4:6443 --token xxxxxxx.xxxxxxxxxxxxxxxxxxx \
--discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

    Step 6 Deploy the Intel SGX Device Plugin for Kubernetes

    Here we would want to deploy the plugin as a DaemonSet, so pull the source code.

    $ git clone https://github.com/intel/intel-device-plugins-for-kubernetes.git

    In the working directory, compile with

    $ make intel-sgx-plugin
$ make intel-sgx-initcontainer

    Deploy the DaemonSet with

    $ kubectl apply -k deployments/sgx_plugin/overlays/epc-register/

    Verify with (replace the with your own node name)

    $ kubectl describe node <node name> | grep sgx.intel.com