Set limits for XCOM pod

[dolfinus]

My k8s namespace has specified ResourceQuota, and thus every container should be created with both requests and limits resources set.

I've tried to run simple dag like this:

import datetime

from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator

from kubernetes.client import models as k8s

with DAG(
    dag_id="example_k8s_operator",
    schedule=None,
    start_date=datetime.datetime(2021, 1, 1),
    catchup=False,
    tags=["example"],
) as dag:

    task = KubernetesPodOperator(
        namespace="onetl-demo-user-namespace",
        name="hello-dry-run",
        image="debian",
        cmds=["bash", "-cx"],
        arguments=["echo", "10"],
        labels={"foo": "bar"},
        task_id="dry_run_demo",
        container_resources=k8s.V1ResourceRequirements(
            requests={"memory": "250M", "cpu": "100m"},
            limits={"memory": "250M", "cpu": "100m"},
        ),
        do_xcom_push=True,
    )

but got an exception:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 135, in run_pod_async
    body=sanitized_pod, namespace=pod.metadata.namespace, **kwargs
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 7356, in create_namespaced_pod
    return self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 7469, in create_namespaced_pod_with_http_info
    collection_formats=collection_formats)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 353, in call_api
    _preload_content, _request_timeout, _host)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 184, in __call_api
    _request_timeout=_request_timeout)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 397, in request
    body=body)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 281, in POST
    body=body)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 234, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"hello-dry-run-8863acf43ec04fa0b293f81ca981d701\" is forbidden: failed quota: portalquota: must specify limits.cpu,limits.memory","reason":"Forbidden","details":{"name":"hello-dry-run-8863acf43ec04fa0b293f81ca981d701","kind":"pods"},"code":403}

Full pod spec from logs:

{
  "apiVersion": "v1",
  "kind": "Pod",
  "metadata": {
    "annotations": {},
    "labels": {
      "foo": "bar",
      "dag_id": "example_k8s_operator",
      "task_id": "dry_run_demo",
      "run_id": "manual__2022-12-05T151152.3955870000-b2ac559bc",
      "kubernetes_pod_operator": "True",
      "try_number": "1",
      "airflow_version": "2.4.1",
      "airflow_kpo_in_cluster": "True"
    },
    "name": "hello-dry-run-8863acf43ec04fa0b293f81ca981d701",
    "namespace": "onetl-demo-user-namespace"
  },
  "spec": {
    "affinity": {},
    "containers": [
      {
        "args": [
          "echo",
          "10"
        ],
        "command": [
          "bash",
          "-cx"
        ],
        "env": [],
        "envFrom": [],
        "image": "debian",
        "name": "base",
        "ports": [],
        "resources": {
          "limits": {
            "memory": "250M",
            "cpu": "100m"
          },
          "requests": {
            "memory": "250M",
            "cpu": "100m"
          }
        },
        "volumeMounts": [
          {
            "mountPath": "/airflow/xcom",
            "name": "xcom"
          }
        ]
      },
      {
        "command": [
          "sh",
          "-c",
          "trap \"exit 0\" INT; while true; do sleep 1; done;"
        ],
        "image": "alpine",
        "name": "airflow-xcom-sidecar",
        "resources": {
          "requests": {
            "cpu": "1m"
          }
        },
        "volumeMounts": [
          {
            "mountPath": "/airflow/xcom",
            "name": "xcom"
          }
        ]
      }
    ],
    "hostNetwork": false,
    "imagePullSecrets": [],
    "initContainers": [],
    "nodeSelector": {},
    "restartPolicy": "Never",
    "securityContext": {},
    "tolerations": [],
    "volumes": [
      {
        "emptyDir": {},
        "name": "xcom"
      }
    ]
  }
}

Container with pod command has the same resources I've set using KubernetesPodOperator.container_resources, but sidecar container with XCOM fetcher is created without limits, causing this exception. As a result I cannot get XCOM of any task created using this operator.

Here I've set limits as 1m for cpu and 10m for memory. IMHO, sleep 1 command does not need more resources, even 10m maybe too much.

I haven't add any options to change these values (like in #26766), it seems that currently there is no way to change command executed by this container.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[uranusjr]

Tests are failing.

[potiuk]

How about fixing the tests @dolfinus ?

[dolfinus]

Is there any way to get failing pod log or something like that? I don't understand why container is starting using one executor but not other ones, and these executors are changing all the time

[potiuk]

Surely - you can download the logs from the job - just click on the "Summary" and you will be able to downlad the logs from failed jobs - it is an artifact that is attached to the job (this is a complete export of all logs from failed test).

Also you can very easily reproduce what the tests are doing locally by following our documentation: https://github.com/apache/airflow/blob/main/TESTING.rst#typical-testing-pattern-for-kubernetes-tests is a step-by-step wizard like guide and above that you will find explanation of what happens in each step. Also breeze has k9s integration included so that you can explore the logs and everything else.

[potiuk]

[dolfinus]

XCOM sidecar container was killed because of OOM, even on memory limit 10Mi and 20Mi. I didn't expect that bash + sleep requires more RAM than 1Mi. Increased limit to 50Mi, fixed.

[dolfinus]

Any updates?

[potiuk]

Well you removed two reviewers that likely would have been much better to review it (not sure why) so you limited your chances to get it reviewed ...

[potiuk]

I am not sure we want to set arbitrary limits for the pods @dstandish @jedcunningham WDYT?

[dolfinus]

Well you removed two reviewers

There was a "Requested changes" status, and I've clicked on it to inform that changes were made. I don't have permissions to change list of reviewers, maybe this is some kind of Github bug.

[potiuk]

Well you removed two reviewers

There was a "Requested changes" status, and I've clicked on it to inform that changes were made. I don't have permissions to change list of reviewers, maybe this is some kind of Github bug.

Maybe. I've added them back.

[potiuk]

Now. There are some confflicts to resolve - It's been some time it was out (and there is some backlog of issues/PRs so this takes longer than usual) - can you please rebase and fix the conflicts though.

I have not looked in detail and we've not heard from @jedcunningham and @dstandish, but I think the numbers here are too magical and forcing them on everyone is a bit too much to solve your case @dolfinus. You can add a feature to optionally add resource quota for those PODS with default being "not set" - that is the best way of handling it. I hardly imagine we will merge a change that we have a fixed memory used for the XCom POD (and it's backwards incompatible too).

[dolfinus]

Ok, got your point.

I've added xcom_sidecar_container_resources key to extras field of k8s connection, just like xcom_sidecar_container_image. User can set here resources in JSON syntax, and they will be passed to sidecar container.

[potiuk]

Thsi is much better. Thank you :)