Skip to content

Commit

Permalink
chore: add info on toleration to be used with daemonsets (#521)
Browse files Browse the repository at this point in the history
* chore: add info on toleration to be used with daemonsets
  • Loading branch information
acarranoqovery authored Feb 4, 2025
1 parent 40828bd commit 3263b12
Show file tree
Hide file tree
Showing 3 changed files with 72 additions and 16 deletions.
44 changes: 36 additions & 8 deletions website/guides/advanced/deploy-daemonset-with-karpenter.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
last_modified_on: "2025-01-08"
last_modified_on: "2025-02-04"
$schema: "/.meta/.schemas/guides.json"
title: Deploy a DaemonSet in a Karpenter context
description: How to ensure your DaemonSet is well deployed when you are using Karpenter.
author_github: https://github.com/baalooos
author_github: https://github.com/acarranoqovery
tags: ["type: tutorial", "technology: qovery", "installation_guide: aws"]
hide_pagination: true
---
Expand All @@ -12,7 +12,9 @@ import Alert from '@site/src/components/Alert';
import Steps from '@site/src/components/Steps';
import Assumptions from '@site/src/components/Assumptions';

[Karpenter](https://karpenter.sh/) is a great way to cut your AWS bill. It provides a simple and flexible way to scale and optimize your resource consumption. But there is a known [issue](https://github.com/kubernetes-sigs/karpenter/issues/731) with capacity planning when deploying DaemonSets. In this guide, we will explore this issue and demonstrate how to resolve it using PriorityClass.
[Karpenter](https://karpenter.sh/) is a great way to cut your AWS bill. It provides a simple and flexible way to scale and optimize your resource consumption.

But when it comes to deploying DaemonSets, you need to take care of some specific configurations to ensure that your DaemonSet is well deployed. This guide will show you how to deploy a DaemonSet in a Karpenter context.

<!--
THIS FILE IS AUTOGENERATED!
Expand All @@ -33,16 +35,15 @@ Key features of DaemonSets include:

This makes DaemonSets a powerful tool for maintaining uniformity and reliability in the operation of essential services across a Kubernetes cluster.

## What is the problem?

## Priority classes

There is a [known issue](https://github.com/kubernetes-sigs/karpenter/issues/731) with Karpenter and DaemonSets when scaling nodes. DaemonSets ensure a copy of a pod runs on every node, consuming additional resources that Karpenter does not consider, leading to potential resource contention and under-provisioned nodes.

This forces operators to over-provision their nodes, resulting in inefficient resource utilization and higher costs. While the Kubernetes community and Karpenter developers are working on solutions, users currently need to manually adjust resource allocations and monitor node utilization to mitigate these issues.

A way to resolve this problem is to use a PriorityClass and attach it to the DaemonSet we are creating.

## How to resolve it?

### What is a PriorityClass?

A PriorityClass in Kubernetes is a resource used to assign priority levels to pods. It helps the Kubernetes scheduler make decisions during resource contention by determining which pods should be scheduled first or evicted in case of resource shortages.
Expand All @@ -66,9 +67,36 @@ Many Helm charts include built-in values to automatically create and configure P
When deploying Qovery on a cluster, a dedicated PriorityClass named `qovery-standard-priority` is created automatically.
You can use this PriorityClass when deploying a new Helm chart or DaemonSet to ensure that DaemonSets deploy correctly, even during resource contention.

## Conclusion

While tools like Karpenter are excellent for optimizing costs and reducing your AWS bill, some manual configuration is sometimes required. This issue demonstrates that adding a single PriorityClass can effectively prevent complex resource allocation problems.
## Targeting all the nodes

When deploying a DaemonSet, you can use taints and affinities to control where the DaemonSet pods are scheduled. This can help you ensure that the DaemonSet pods are deployed only on nodes that are interesting for you (for monitoring, logging etc..).

### Nodepool default taints

By default, 2 nodepools are deployed with Karpenter: `default` and `stable`. On the `stable` nodepool, a taint has been defined to ensure that only pods having a toleration with the `stable` nodepool can be scheduled on it (key `nodepool/stable`).

### How to target every node

To ensure that the DaemonSet pods are scheduled on every node, you can add a toleration to the DaemonSet pods that matches any taint. You also need to add an affinity to the DaemonSet pods to ensure that they are not scheduled on Fargate nodes.

Here's an example of how you can do this with a helm chart:

```yaml

tolerations:
- operator: Exists
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: eks.amazonaws.com/compute-type
operator: NotIn
values:
- fargate

```



42 changes: 35 additions & 7 deletions website/guides/advanced/deploy-daemonset-with-karpenter.md.erb
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
$schema: "/.meta/.schemas/guides.json"
title: Deploy a DaemonSet in a Karpenter context
description: How to ensure your DaemonSet is well deployed when you are using Karpenter.
author_github: https://github.com/baalooos
author_github: https://github.com/acarranoqovery
tags: ["type: tutorial", "technology: qovery", "installation_guide: aws"]
hide_pagination: true
---
Expand All @@ -11,7 +11,9 @@ import Alert from '@site/src/components/Alert';
import Steps from '@site/src/components/Steps';
import Assumptions from '@site/src/components/Assumptions';

[Karpenter](https://karpenter.sh/) is a great way to cut your AWS bill. It provides a simple and flexible way to scale and optimize your resource consumption. But there is a known [issue](https://github.com/kubernetes-sigs/karpenter/issues/731) with capacity planning when deploying DaemonSets. In this guide, we will explore this issue and demonstrate how to resolve it using PriorityClass.
[Karpenter](https://karpenter.sh/) is a great way to cut your AWS bill. It provides a simple and flexible way to scale and optimize your resource consumption.

But when it comes to deploying DaemonSets, you need to take care of some specific configurations to ensure that your DaemonSet is well deployed. This guide will show you how to deploy a DaemonSet in a Karpenter context.

## What is a DaemonSet?

Expand All @@ -24,16 +26,15 @@ Key features of DaemonSets include:

This makes DaemonSets a powerful tool for maintaining uniformity and reliability in the operation of essential services across a Kubernetes cluster.

## What is the problem?

## Priority classes

There is a [known issue](https://github.com/kubernetes-sigs/karpenter/issues/731) with Karpenter and DaemonSets when scaling nodes. DaemonSets ensure a copy of a pod runs on every node, consuming additional resources that Karpenter does not consider, leading to potential resource contention and under-provisioned nodes.

This forces operators to over-provision their nodes, resulting in inefficient resource utilization and higher costs. While the Kubernetes community and Karpenter developers are working on solutions, users currently need to manually adjust resource allocations and monitor node utilization to mitigate these issues.

A way to resolve this problem is to use a PriorityClass and attach it to the DaemonSet we are creating.

## How to resolve it?

### What is a PriorityClass?

A PriorityClass in Kubernetes is a resource used to assign priority levels to pods. It helps the Kubernetes scheduler make decisions during resource contention by determining which pods should be scheduled first or evicted in case of resource shortages.
Expand All @@ -57,6 +58,33 @@ Many Helm charts include built-in values to automatically create and configure P
When deploying Qovery on a cluster, a dedicated PriorityClass named `qovery-standard-priority` is created automatically.
You can use this PriorityClass when deploying a new Helm chart or DaemonSet to ensure that DaemonSets deploy correctly, even during resource contention.

## Conclusion

While tools like Karpenter are excellent for optimizing costs and reducing your AWS bill, some manual configuration is sometimes required. This issue demonstrates that adding a single PriorityClass can effectively prevent complex resource allocation problems.
## Targeting all the nodes

When deploying a DaemonSet, you can use taints and affinities to control where the DaemonSet pods are scheduled. This can help you ensure that the DaemonSet pods are deployed only on nodes that are interesting for you (for monitoring, logging etc..).

### Nodepool default taints

By default, 2 nodepools are deployed with Karpenter: `default` and `stable`. On the `stable` nodepool, a taint has been defined to ensure that only pods having a toleration with the `stable` nodepool can be scheduled on it (key `nodepool/stable`).

### How to target every node

To ensure that the DaemonSet pods are scheduled on every node, you can add a toleration to the DaemonSet pods that matches any taint. You also need to add an affinity to the DaemonSet pods to ensure that they are not scheduled on Fargate nodes.

Here's an example of how you can do this with a helm chart:

```yaml

tolerations:
- operator: Exists
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: eks.amazonaws.com/compute-type
operator: NotIn
values:
- fargate

```
2 changes: 1 addition & 1 deletion website/metadata.js
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,7 @@ module.exports = {
"title": "Deploy Frontend App"
},
{
"author_github": "https://github.com/baalooos",
"author_github": "https://github.com/acarranoqovery",
"description": null,
"id": "/advanced/deploy-daemonset-with-karpenter",
"last_modified_on": null,
Expand Down

0 comments on commit 3263b12

Please sign in to comment.