Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add prometheus-operator tutorial #524

Merged
merged 4 commits into from
Mar 27, 2020
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
184 changes: 184 additions & 0 deletions docs/gitbook/tutorials/prometheus-operator.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
# Flagger with Prometheus Operator

This guide will show you how to use Flagger and Prometheus Operator.
This guide will handle only Blue/Green Deployment with podinfo application

## Prerequisites

Flagger and Prometheus Operator requires a Kubernetes cluster **v1.11** or newer

Install Prometheus-Operator with Helm v3:

```bash
helm repo add stable https://kubernetes-charts.storage.googleapis.com
helm repo update
kubectl create ns monitoring
helm upgrade -i prometheus stable/prometheus-operator \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use fullnameOverride and set the namespace to monitoring so that the URL will look better.

--namespace monitoring \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
--set fullnameOverride=prometheus
```

The `prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false` option allows Prometheus-Operator to watch serviceMonitor outside of his namespace.

You can also set `prometheus.service.type=nodePort` if you want to have access the Prometheus UI

Install Flagger with Helm v3:

```bash
helm repo add flagger https://flagger.app
helm repo update
kubectl create ns flagger
helm upgrade -i flagger flagger/flagger \
--namespace flagger \
--set metricsServer=http://prometheus-prometheus.monitoring:9090 \
--set meshProvider=kubernetes
```

The `meshProvider` option can be changed to your value, if you want to do something else than Blue/Green Deployment

Install Flagger Loadtester with Helm v3:

```bash
helm repo add flagger https://flagger.app
helm repo update
kubectl create ns flagger
helm upgrade -i loadtester flagger/loadtester \
--namespace flagger
```

Install podinfo with Helm v3:

```bash
helm repo add sp https://stefanprodan.github.io/podinfo
helm repo update
kubectl create ns test
helm upgrade -i podinfo sp/podinfo \
--namespace test
```

## Setting ServiceMonitor

Prometheus Operator is using mostly serviceMonitor instead of annotations.
In order to catch metrics for primary and canary service, you will need to create 2 serviceMonitors :

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: podinfo
namespace: test
spec:
endpoints:
- path: /metrics
port: http
interval: 15s
selector:
matchLabels:
app: podinfo
```

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: podinfo-canary
namespace: test
spec:
endpoints:
- path: /metrics
port: http
interval: 15s
selector:
matchLabels:
app: podinfo-canary
```

We are setting `interval: 15s` to have a more aggressive scraping
If you do not define it, you must to use a longer interval in the Canary object

## Setting Custom metrics

Prometheus Operator is relabeling for every serviceMonitor, you can create custom metrics to you own filter.

```yaml
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
name: request-success-rate
namespace: test
spec:
provider:
address: http://prometheus-prometheus.monitoring:9090
type: prometheus
query: |
rate(
http_requests_total{
namespace="{{ namespace }}",
job="{{ target }}-canary",
status!~"5.*"
}[{{ interval }}])
/
rate(
http_requests_total{
namespace="{{ namespace }}",
job="{{ target }}-canary"
}[{{ interval }}]
) * 100
```

You can also use `pod="{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"` instead of `job={{ target }}-canary`, if you want.

## Creating Canary

```yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
provider: kubernetes
targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
progressDeadlineSeconds: 60
service:
port: 9898
portDiscovery: true
analysis:
interval: 30s
iterations: 10
threshold: 2
metrics:
- name: http-success-rate
templateRef:
name: request-success-rate
namespace: test
thresholdRange:
min: 99
interval: 1m
webhooks:
- name: smoke-test
type: pre-rollout
url: "http://loadtester.flagger/"
timeout: 15s
metadata:
type: bash
cmd: "curl -sd 'anon' http://podinfo-canary.test:9898/token | grep token"
- name: load-test
type: rollout
url: "http://loadtester.flagger/"
timeout: 5s
metadata:
type: cmd
cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898"
```

## Test the canary

Execute `kubectl -n test set image deployment/podinfo podinfo=stefanprodan/podinfo:3.1.0` to see if everything works