Move CNCF Kubernetes to new provider structure

[jason810496]

related: #46045

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[rawwar]

@hardeybisey is already working on this - #46045 (comment)

Just noticed the comment - #46132 (comment)

[bugraoz93]

One of the docs is failing due to a path showing old system tests. apache-airflow/tutorial/taskflow.rst

[jason810496]

The remain errors are caused by tests/always/test_project_structure.py::TestCncfProviderProjectStructure::test_missing_examples and import error for airflow.providers.cncf in kubernetes_tests/test_kubernetes_pod_operator.py.

kubernetes_tests/test_kubernetes_pod_operator.py:39: in <module>
    from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
E   ModuleNotFoundError: No module named 'airflow.providers.cncf'

I tried updating the path in tests/always/test_project_structure.py, replacing providers/tests/cncf/kubernetes/ with providers/cncf/kubernetes/tests/provider_tests/cncf/kubernetes/, and ran the test locally, but it still fails.

I'm also unsure why the import in kubernetes_tests/test_kubernetes_pod_operator.py is failing, might need some hints on how to fix these errors. 🙏

[rawwar]

@jason810496 , which failing test has this import error? I checked failing tests and don't see any import errors. Can you share the link to the failing test? or is it happening locally?

[jason810496]

@jason810496 , which failing test has this import error? I checked failing tests and don't see any import errors. Can you share the link to the failing test? or is it happening locally?

All the k8s tests encounter the import error. e.g. https://github.com/apache/airflow/actions/runs/13149821313/job/36695881525?pr=46436

[rawwar]

@jason810496 , which failing test has this import error? I checked failing tests and don't see any import errors. Can you share the link to the failing test? or is it happening locally?

All the k8s tests encounter the import error. e.g. https://github.com/apache/airflow/actions/runs/13149821313/job/36695881525?pr=46436

Ah sorry. I just realised you mentioned about two issues. I resolved the issue with example_dag in google provider. looking into it. Not sure about k8s tests. Will look at it after example dag tests

[rawwar]

@jason810496 , I noticed that adding

MISSING_EXAMPLES_FOR_CLASSES = {
        "src.airflow.providers.cncf.kubernetes.operators.job.KubernetesDeleteJobOperator"
    }

to TestCncfProviderProjectStructure and that's ignoring the issues. But, in actual, test for the class is present at airflow/providers/cncf/kubernetes/tests/provider_tests/cncf/kubernetes/operators/test_job.py

So, issue is definitely with the src prefix in the import path. That's a direction to look into?

EDIT: I suspect issue is because of the way we find example dags at https://github.com/rawwar/airflow/blob/4821da5cc3fe58e1f64d49967b52026adfa89dbe/tests/always/test_project_structure.py#L307

[rawwar]

Ok, so, the issue is that PROVIDER is set to cncf in https://github.com/rawwar/airflow/blob/4821da5cc3fe58e1f64d49967b52026adfa89dbe/tests/always/test_project_structure.py#L288

because of that, its unable to find the example dags

[jason810496]

Ok, so, the issue is that PROVIDER is set to cncf in https://github.com/rawwar/airflow/blob/4821da5cc3fe58e1f64d49967b52026adfa89dbe/tests/always/test_project_structure.py#L288

because of that, its unable to find the example dags

I just replaced the PROVIDER = "cncf" with PROVIDER = "cncf/kubernetes" and ran the test, but it still fail.
https://github.com/rawwar/airflow/blob/4821da5cc3fe58e1f64d49967b52026adfa89dbe/tests/always/test_project_structure.py#L571

UPDATED:
The test fail due to:

{'src.airflow.providers.cncf.kubernetes.operators.spark_kubernetes.SparkKubernetesOperator': <ast.ClassDef object at 0xffffb0c46910>}
Classes with missing examples:
    src.airflow.providers.cncf.kubernetes.operators.job.KubernetesDeleteJobOperator
    src.airflow.providers.cncf.kubernetes.operators.job.KubernetesJobOperator
    src.airflow.providers.cncf.kubernetes.operators.job.KubernetesPatchJobOperator
    src.airflow.providers.cncf.kubernetes.operators.kueue.KubernetesInstallKueueOperator
    src.airflow.providers.cncf.kubernetes.operators.kueue.KubernetesStartKueueJobOperator
    src.airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator
    src.airflow.providers.cncf.kubernetes.operators.resource.KubernetesCreateResourceOperator
    src.airflow.providers.cncf.kubernetes.operators.resource.KubernetesDeleteResourceOperator
    src.airflow.providers.cncf.kubernetes.operators.resource.KubernetesResourceBaseOperator
    src.airflow.providers.cncf.kubernetes.operators.spark_kubernetes.SparkKubernetesOperator
    src.airflow.providers.cncf.kubernetes.sensors.spark_kubernetes.SparkKubernetesSensor
FAILED

I think by adding MISSING_EXAMPLES_FOR_CLASSES set will resolve this error.

[rawwar]

I think by adding MISSING_EXAMPLES_FOR_CLASSES set will resolve this error.

I was actually checking. I see example tests at airflow/providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_job.py.

Somehow, these aren't being imported

EDIT: Issue is definitely because of https://github.com/rawwar/airflow/blob/4821da5cc3fe58e1f64d49967b52026adfa89dbe/tests/always/test_project_structure.py#L243

Only for this provider, its prefixed with src.. I checked google tests and I don't see src. being prefixed. fixing that should resolve the issues

[rawwar]

So, the actual issue for the example test is at https://github.com/rawwar/airflow/blob/4821da5cc3fe58e1f64d49967b52026adfa89dbe/tests/always/test_project_structure.py#L272

Here we just split and consider from 2nd index. since, its cncf.kubernetes.src in this case, its failing. This needs to be fixed. Maybe later or in this PR

On to K8s failing tests

[rawwar]

@jason810496 , raised a separate pr - #46454 . that might handle it. Locally, tests passed . looking into k8s tests failure

[rawwar]

it looks like #46454 also fixes K8s tests. I just ran them locally and they passed

[jason810496]

it looks like #46454 also fixes K8s tests. I just ran them locally and they passed

If then, I will revert the last commit and wait for #46454 be merged. Big thanks to @rawwar !

[rawwar]

Can you pull the changes from the PR? these k8s tests didn't run on the other PR. Once you pull, we can at least confirm that the changes fix the k8s tests.

[rawwar]

So, the only other fix required here for the example tests is to make providers = "cncf/kubernetes".

But, k8s tests are still failing. I just raised a dummy pr to see the test results and they failed. Locally, They are successful.

[potiuk]

@jason810496 -> the k8s tests here are failing because you need to modify scripts/ci/kubernetes/k8s_requirements.txt -> You need to add this:

-e ./providers/cncf/kubernetes

[jason810496]

The import error has finally been resolved! Big thanks to @potiuk!

The only remaining CI failures:

• Additional PROD image tests
  • Docker Compose quick start with PROD image
    • upstream_failed in docker_tests/test_docker_compose_quick_start.py::test_trigger_dag_and_wait_for_result
• K8s tests
  • upstream_failed or failed for both CeleryExecutor and LocalExecutor.

Not sure if rerunning the tests would resolve these errors.

[potiuk]

Not sure if rerunning the tests would resolve these errors

Those are tests failing in main now - due to some task_sdk errors and there are other PRs that should already fix them #46492 - merged after your change, so rebasing should fix them. Let me try it.

[potiuk]

Ah... You need to rebase :( there are some changes merged in the meantime. Sorry :(

But we are REALLY close.

[jason810496]

But we are REALLY close.

Hope this one will be the last one to be green 🚀 🟢

[potiuk]

Ah.. For some reason you have all teh _api files generated added to that PR @jason810496 -> you should remove them!

[jason810496]

Ah.. For some reason you have all teh _api files generated added to that PR @jason810496 -> you should remove them!

Do you mean all the docs prefix with _api should be removed ?

[potiuk]

Yep. I have not seen you are around - just pushed that fixup.

Yeah. The _api docs are generated automatically when docs are being built and they should be .gitignored in general - but they were added in this PR.

[potiuk]

Wooho.

[jason810496]

Wooho.

Greatly appreciate it, thanks @potiuk! 🙌🎉