Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate deployment manifests from helm chart #475

Merged
merged 10 commits into from
May 4, 2020
Merged
1 change: 1 addition & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ script:
- make
- go test -covermode=count -coverprofile=profile.cov ./pkg/...
- $GOPATH/bin/goveralls -coverprofile=profile.cov -service=travis-ci
- make kubeval
49 changes: 44 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -25,25 +25,43 @@ GOBIN=$(shell pwd)/bin

.EXPORT_ALL_VARIABLES:

bin/aws-ebs-csi-driver:
mkdir -p bin
bin/aws-ebs-csi-driver: | bin
CGO_ENABLED=0 GOOS=linux go build -ldflags ${LDFLAGS} -o bin/aws-ebs-csi-driver ./cmd/

bin/mockgen:
bin /tmp/helm /tmp/kubeval:
@mkdir -p $@

bin/helm: | /tmp/helm bin
@curl -o /tmp/helm/helm.tar.gz -sSL https://get.helm.sh/helm-v3.1.2-linux-amd64.tar.gz
@tar -zxf /tmp/helm/helm.tar.gz -C bin --strip-components=1
@rm -rf /tmp/helm/*

bin/kubeval: | /tmp/kubeval bin
@curl -o /tmp/kubeval/kubeval.tar.gz -sSL https://github.com/instrumenta/kubeval/releases/download/0.15.0/kubeval-linux-amd64.tar.gz
@tar -zxf /tmp/kubeval/kubeval.tar.gz -C bin kubeval
@rm -rf /tmp/kubeval/*

bin/mockgen: | bin
go get github.com/golang/mock/mockgen@latest

bin/golangci-lint:
bin/golangci-lint: | bin
echo "Installing golangci-lint..."
curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh| sh -s v1.21.0

.PHONY: kubeval
kubeval: bin/kubeval
bin/kubeval -d deploy/kubernetes/base,deploy/kubernetes/cluster,deploy/kubernetes/overlays -i kustomization.yaml,crd_.+\.yaml,controller_add

mockgen: bin/mockgen
./hack/update-gomock

.PHONY: verify
verify: bin/golangci-lint
echo "Running golangci-lint..."
./bin/golangci-lint run --deadline=10m
echo "Congratulations! All Go source files have been linted."

.PHONY: test
test:
go test -v -race ./cmd/... ./pkg/...

Expand All @@ -52,7 +70,7 @@ test-sanity:
#go test -v ./tests/sanity/...
echo "succeed"

bin/k8s-e2e-tester:
bin/k8s-e2e-tester: | bin
go get github.com/aws/aws-k8s-tester/e2e/tester/cmd/k8s-e2e-tester@master

.PHONY: test-e2e-single-az
Expand Down Expand Up @@ -84,3 +102,24 @@ push-release:
.PHONY: push
push:
docker push $(IMAGE):latest

.PHONY: generate-kustomize
generate-kustomize: bin/helm
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a bit over complicated. Is it possible to simply this step? I imagine it will be hard to update this list of command, whenever there is some changes or refactoring to the helm templates. I think it is okay to restructure the layer out of kustomize file structure to make the generation earlier and more maintainable. Maybe combine some of the kustomize files into one for each layer? As they are all auto generated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I don't like it much either. What I ran into is that when running helm template for files with more than one resource those resources weren't rendered in a consistent order. I could run the target twice in a row and get the same resources but in a different order which makes it difficult to manage in source control. I would be open if anyone has a way to produce reliably reproducible output and keep the alpha features separate in the folder structure.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the limitation with helm. I think I can live with it for now, and we can follow up with this as a separate issue.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed this. It should have been fixed in Helm 3.1.0: helm/helm#6842. Weird you were seeing this with Helm 3.1.2 here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I saw that helm issue when I was working on this but still saw issues with resources moving around in the processed output on multiple runs without changing anything.

cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/clusterrole-attacher.yaml > ../deploy/kubernetes/base/clusterrole-attacher.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/clusterrole-provisioner.yaml > ../deploy/kubernetes/base/clusterrole-provisioner.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/clusterrolebinding-attacher.yaml > ../deploy/kubernetes/base/clusterrolebinding-attacher.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/clusterrolebinding-provisioner.yaml > ../deploy/kubernetes/base/clusterrolebinding-provisioner.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/controller.yaml -f ../deploy/kubernetes/values/controller.yaml > ../deploy/kubernetes/base/controller.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/csidriver.yaml > ../deploy/kubernetes/base/csidriver.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/node.yaml -f ../deploy/kubernetes/values/controller.yaml > ../deploy/kubernetes/base/node.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/serviceaccount-csi-controller.yaml > ../deploy/kubernetes/base/serviceaccount-csi-controller.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/clusterrole-resizer.yaml -f ../deploy/kubernetes/values/resizer.yaml > ../deploy/kubernetes/overlays/alpha/rbac_add_resizer_clusterrole.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/clusterrole-snapshot-controller.yaml -f ../deploy/kubernetes/values/snapshotter.yaml > ../deploy/kubernetes/overlays/alpha/rbac_add_snapshot_controller_clusterrole.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/clusterrole-snapshotter.yaml -f ../deploy/kubernetes/values/snapshotter.yaml > ../deploy/kubernetes/overlays/alpha/rbac_add_snapshotter_clusterrole.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/clusterrolebinding-resizer.yaml -f ../deploy/kubernetes/values/resizer.yaml > ../deploy/kubernetes/overlays/alpha/rbac_add_resizer_clusterrolebinding.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/clusterrolebinding-snapshot-controller.yaml -f ../deploy/kubernetes/values/snapshotter.yaml > ../deploy/kubernetes/overlays/alpha/rbac_add_snapshot_controller_clusterrolebinding.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/clusterrolebinding-snapshotter.yaml -f ../deploy/kubernetes/values/snapshotter.yaml > ../deploy/kubernetes/overlays/alpha/rbac_add_snapshotter_clusterrolebinding.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/role-snapshot-controller-leaderelection.yaml -f ../deploy/kubernetes/values/snapshotter.yaml > ../deploy/kubernetes/overlays/alpha/rbac_add_snapshot_controller_leaderelection_role.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/rolebinding-snapshot-controller-leaderelection.yaml -f ../deploy/kubernetes/values/snapshotter.yaml > ../deploy/kubernetes/overlays/alpha/rbac_add_snapshot_controller_leaderelection_rolebinding.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/serviceaccount-snapshot-controller.yaml -f ../deploy/kubernetes/values/snapshotter.yaml > ../deploy/kubernetes/overlays/alpha/serviceaccount-snapshot-controller.yaml
cd aws-ebs-csi-driver && ../bin/helm template kustomize . -s templates/statefulset.yaml -f ../deploy/kubernetes/values/snapshotter.yaml > ../deploy/kubernetes/overlays/alpha/snapshot_controller.yaml
2 changes: 1 addition & 1 deletion aws-ebs-csi-driver/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v1
appVersion: "0.5.0"
name: aws-ebs-csi-driver
description: A Helm chart for AWS EBS CSI Driver
version: 0.3.0
version: 0.4.0
kubeVersion: ">=1.13.0-0"
home: https://github.com/kubernetes-sigs/aws-ebs-csi-driver
sources:
Expand Down
17 changes: 14 additions & 3 deletions aws-ebs-csi-driver/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,24 @@ Create chart name and version as used by the chart label.
Common labels
*/}}
{{- define "aws-ebs-csi-driver.labels" -}}
app.kubernetes.io/name: {{ include "aws-ebs-csi-driver.name" . }}
{{ include "aws-ebs-csi-driver.selectorLabels" . }}
{{- if ne .Release.Name "kustomize" }}
helm.sh/chart: {{ include "aws-ebs-csi-driver.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{- end -}}

{{/*
Common selector labels
*/}}
{{- define "aws-ebs-csi-driver.selectorLabels" -}}
app.kubernetes.io/name: {{ include "aws-ebs-csi-driver.name" . }}
{{- if ne .Release.Name "kustomize" }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{- end -}}

{{/*
Expand All @@ -53,6 +64,6 @@ Convert the `--extra-volume-tags` command line arg from a map.
{{- $noop := printf "%s=%s" $key $value | append $result.pairs | set $result "pairs" -}}
{{- end -}}
{{- if gt (len $result.pairs) 0 -}}
- --extra-volume-tags={{- join "," $result.pairs -}}
{{- printf "%s=%s" "- --extra-volume-tags" (join "," $result.pairs) -}}
{{- end -}}
{{- end -}}
20 changes: 20 additions & 0 deletions aws-ebs-csi-driver/templates/clusterrole-attacher.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-external-attacher-role
labels:
{{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csinodeinfos"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update"]
35 changes: 35 additions & 0 deletions aws-ebs-csi-driver/templates/clusterrole-provisioner.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-external-provisioner-role
labels:
{{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["get", "list"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
{{- if .Values.enableVolumeResizing }}
---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-external-resizer-role
labels:
{{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
rules:
# The following rule should be uncommented for plugins that require secrets
# for provisioning.
Expand All @@ -26,18 +28,4 @@ rules:
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-csi-resizer-binding
subjects:
- kind: ServiceAccount
name: ebs-csi-controller-sa
namespace: kube-system
roleRef:
kind: ClusterRole
name: ebs-external-resizer-role
apiGroup: rbac.authorization.k8s.io

{{- end}}
35 changes: 35 additions & 0 deletions aws-ebs-csi-driver/templates/clusterrole-snapshot-controller.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{{- if .Values.enableVolumeSnapshot }}
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-snapshot-controller-role
labels:
{{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents"]
verbs: ["create", "get", "list", "watch", "update", "delete"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots/status"]
verbs: ["update"]

{{- end }}
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
{{- if .Values.enableVolumeSnapshot }}
---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-external-snapshotter-role
labels:
{{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
rules:
- apiGroups: [""]
resources: ["events"]
Expand All @@ -20,19 +22,4 @@ rules:
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents/status"]
verbs: ["update"]
---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-csi-snapshotter-binding
subjects:
- kind: ServiceAccount
name: ebs-csi-controller-sa
namespace: kube-system
roleRef:
kind: ClusterRole
name: ebs-external-snapshotter-role
apiGroup: rbac.authorization.k8s.io


{{- end }}
15 changes: 15 additions & 0 deletions aws-ebs-csi-driver/templates/clusterrolebinding-attacher.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-csi-attacher-binding
labels:
{{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
subjects:
- kind: ServiceAccount
name: ebs-csi-controller-sa
namespace: kube-system
roleRef:
kind: ClusterRole
name: ebs-external-attacher-role
apiGroup: rbac.authorization.k8s.io
15 changes: 15 additions & 0 deletions aws-ebs-csi-driver/templates/clusterrolebinding-provisioner.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-csi-provisioner-binding
labels:
{{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
subjects:
- kind: ServiceAccount
name: ebs-csi-controller-sa
namespace: kube-system
roleRef:
kind: ClusterRole
name: ebs-external-provisioner-role
apiGroup: rbac.authorization.k8s.io
18 changes: 18 additions & 0 deletions aws-ebs-csi-driver/templates/clusterrolebinding-resizer.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{{- if .Values.enableVolumeResizing }}
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-csi-resizer-binding
labels:
{{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
subjects:
- kind: ServiceAccount
name: ebs-csi-controller-sa
namespace: kube-system
roleRef:
kind: ClusterRole
name: ebs-external-resizer-role
apiGroup: rbac.authorization.k8s.io

{{- end}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{{- if .Values.enableVolumeSnapshot }}
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-csi-snapshot-controller-binding
labels:
{{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
subjects:
- kind: ServiceAccount
name: ebs-snapshot-controller
namespace: kube-system
roleRef:
kind: ClusterRole
name: ebs-snapshot-controller-role
apiGroup: rbac.authorization.k8s.io

{{- end }}
18 changes: 18 additions & 0 deletions aws-ebs-csi-driver/templates/clusterrolebinding-snapshotter.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{{- if .Values.enableVolumeSnapshot }}
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ebs-csi-snapshotter-binding
labels:
{{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
subjects:
- kind: ServiceAccount
name: ebs-csi-controller-sa
namespace: kube-system
roleRef:
kind: ClusterRole
name: ebs-external-snapshotter-role
apiGroup: rbac.authorization.k8s.io

{{- end }}
Loading