Added job_clusters as a templated parameter to CreateDatabricksWorkflowOperator

[kunaljubce]

Reverts #45035, which had removed the changes done in #42438. This PR reinstates those changes back as the CI failure issues caused after merging #42438 were unrelated to the change.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[arollet-decathlon]

Hello everyone,
It would be great to see this PR merged, especially since it looks like this change is documented in the provider's Changelog but is not present in codebase: https://github.com/apache/airflow/blob/main/providers/src/airflow/providers/databricks/CHANGELOG.rst?plain=1#L58

Good day

[kunaljubce]

@potiuk Can we merge this please?

[karthi-keyan-n]

Hi folks, is there any view on the release plan for this fix? I'm currently using 2.10.4 and I see the changes has not been tagged with 2.10.5rc1.

[potiuk]

Hi folks, is there any view on the release plan for this fix? I'm currently using 2.10.4 and I see the changes has not been tagged with 2.10.5rc1.

No - they are not. They are part of the providers's release. Providers are released independently from Airfow and you can uprade it on your onw. In fact there is nothing to prevent you to build a dev/pre-release version of databricks provider yourself, install it in your environment, try it out and provide feedback. It's super easy:

1. checkout airflow repo
2. install breeze: uv pip install -e ./dev/breeze
3. breeze release-management prepare-provider-packages databricks --version-suffix-for-pypi dev0
4. Install resulting package that will be placed in dist.

Can you please test it and see if it works - and then when we come to releasing providers (which will likely happen in the next few weeks) - you willbe able to confirm and test RC candidate - when we ask here and vote +1 on the release.

Can we count on your help there?

[potiuk]

This is how it works for me:

[karthi-keyan-n]

checkout airflow repo
install breeze: uv pip install -e ./dev/breeze
breeze release-management prepare-provider-packages databricks --version-suffix-for-pypi dev0
Install resulting package that will be placed in dist.

Hey @potiuk I followed your steps to install and validate. But got stuck post that.

1. I executed the commands which you shared and it generate .whl file in dist directory. dist/apache_airflow_providers_databricks-7.0.0.dev0-py3-none-any.whl
2. I googled and found the command pip install dist/apache_airflow_providers_databricks-7.0.0.dev0-py3-none-any.whl to install the wheel packages.
3. I executed the command in the same directory (airflow source code) and I guess it install airflow there. (This is my assumption)

Other things which I tried was,

1. Build a docker container in local and tried to substitute it with the apache/airflow docker image in my custom airflow code to bring up the airflow env. Post this, I triggered the DAG and I saw only notebook params field in rendered template section.

2. I did some digging and went through the docker file and found ARG AIRFLOW_VERSION="2.10.4" in docker file.
3. I also went through the INSTALL doc within the airflow source code repo to bring up the test env.

If you don't mind, would you able to point me to the right direction from where I can setup in my local and validate? It would be great if you can share the doc for substituting the official airflow image with the custom build image from source code which will make the task earlier to quickly validate