Status of testing Providers that were prepared on January 14, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 7.1.0rc1

• Add configuration_overrides to templated fields (#28920): @tlinkin
• Add a new SSM hook and use it in the System Test context builder (#28755): @ferruzzi
• Add waiter config params to emr.add_job_flow_steps (#28464): @ShelRoman
• Add AWS Sagemaker Auto ML operator and sensor (#28472): @vandonr-amz
• new operator to create a sagemaker experiment (#28837): @vandonr-amz
• Avoid circular import from S3HookUriParseFailure (#28908): @dstandish
• Use compat for cached_property in AWS Batch modules (#28835): @josh-fell
• Apply "unify bucket and key" before "provide bucket" (#28710): @dstandish
• Update S3ToRedshift Operator docs to indicate multiple key functionality (#28705): @RachitSharma2001
• Refactor waiter function and improve unit tests (#28753): @vincbeck
• Better exception raised in case of numpy missing (#28722): @potiuk
• Don't call get_connection from provide_bucket_name (#28716): @dstandish

Provider apache.beam: 4.1.1rc1

• Ensure Beam Go file downloaded from GCS still exists when referenced (#28664): @johannaojeling

Provider apache.hive: 5.1.1rc1

• Move local_infile option from extra to hook parameter (#28811): @potiuk

Provider cncf.kubernetes: 5.1.1rc1

• Fix Incorrect await_container_completion (#28771): @csm10495

Provider common.sql: 1.3.3rc1

• Handle non-compliant behaviour of Exasol cursor (#28744): @potiuk

Provider dbt.cloud: 2.3.1rc1

• Use entire tenant domain name in dbt Cloud connection (#28890): @josh-fell

Provider docker: 3.4.1rc1

• Add correct widgets in Docker Hook (#28700): @potiuk
• make docker operators always use DockerHook for API calls (#28363): @Taragolis

Provider elasticsearch: 4.3.3rc1

• Allow nested attr in elasticsearch host_field (#28878): @dstandish

Provider exasol: 4.1.3rc1

• Handle non-compliant behaviour of Exasol cursor (#28744): @potiuk

Provider google: 8.8.0rc1

• Add deferrable GCSObjectExistenceSensorAsync (#28763): @rajaths010494
• Support partition_columns in BaseSQLToGCSOperator (#28677): @vchiapaikeo
• BigQueryCreateExternalTableOperator fix field delimiter not working with csv (#28856): @fhaubner
• replace private _get_credentials by public get_credentials (#28588): @zazaho
• FixGoogleCampaignManagerReportSensor with QUEUED status (#28735): @kacpermuda
• Fix BigQueryColumnCheckOperator runtime error (#28796): @vchiapaikeo
• Assign datasetReference by default in create_empty_dataset of BigQueryHook (#28782): @shahar1

Provider http: 4.1.1rc1

• Change logging for HttpHook to debug (#28911): @amoghrajesh

Provider jenkins: 3.2.0rc1

• check jenkins result state (#28659): @tnyz

Provider microsoft.azure: 5.1.0rc1

• Hook for managing directories and files in Azure Data Lake Storage Gen2 (#28262): @bharanidharan14
• Hide extra field in Azure Blob Storage wasb (#28914): @IAL32

Provider microsoft.psrp: 2.2.0rc1

• Add option to add arguments to PSRP hook and operator (#27689): @malthe

Provider mongo: 3.1.1rc1

• MongoHook optimization (#28675): @stamixthereal

Provider mysql: 4.0.0rc1

• Move local_infile option from extra to hook parameter (#28811): @potiuk

Provider slack: 7.2.0rc1

• Add general-purpose "notifier" concept to DAGs (#28569): @ephraimbuddy
  Note: we decided to release Slack tough this specific feature is can be used only with Airflow 2.6 (see discussion)

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs:
@ShelRoman @stamixthereal @csm10495 @kacpermuda @Taragolis @shahar1 @ferruzzi @vincbeck @malthe @josh-fell @IAL32 @tlinkin @vchiapaikeo @vandonr-amz @bharanidharan14 @tnyz @rajaths010494 @amoghrajesh

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[amoghrajesh]

@eladkal #28911 has been tested for the RC build. It is working as expected

[vchiapaikeo]

#28677 and #28796 also working fine in the latest RC build.

[stamixthereal]

#28675 has been tested, and it is working as expected

[johannaojeling]

#28664 has been tested with the RC version and is working as expected

[tlinkin]

#28920 test and working as expected

[rajaths010494]

#28763 tested and working as expected

[malthe]

#27689 tested and working as expected.

…

On Sat, 14 Jan 2023 at 13:25, eladkal ***@***.***> wrote: Body I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers? Let us know in the comment, whether the issue is addressed. Those are providers that require testing as there were some substantial changes introduced: Provider amazon: 7.1.0rc1 <https://pypi.org/project/apache-airflow-providers-amazon/7.1.0rc1> - Add configuration_overrides to templated fields (#28920) <#28920>: @tlinkin <https://github.com/tlinkin> - Add a new SSM hook and use it in the System Test context builder (#28755) <#28755>: @ferruzzi <https://github.com/ferruzzi> - Add waiter config params to emr.add_job_flow_steps (#28464) <#28464>: @ShelRoman <https://github.com/ShelRoman> - Add AWS Sagemaker Auto ML operator and sensor (#28472) <#28472>: @vandonr-amz <https://github.com/vandonr-amz> - new operator to create a sagemaker experiment (#28837) <#28837>: @vandonr-amz <https://github.com/vandonr-amz> - Avoid circular import from S3HookUriParseFailure (#28908) <#28908>: @dstandish <https://github.com/dstandish> - Use compat for cached_property in AWS Batch modules (#28835) <#28835>: @josh-fell <https://github.com/josh-fell> - Apply "unify bucket and key" before "provide bucket" (#28710) <#28710>: @dstandish <https://github.com/dstandish> - Update S3ToRedshift Operator docs to indicate multiple key functionality (#28705) <#28705>: @RachitSharma2001 <https://github.com/RachitSharma2001> - Refactor waiter function and improve unit tests (#28753) <#28753>: @vincbeck <https://github.com/vincbeck> - Better exception raised in case of numpy missing (#28722) <#28722>: @potiuk <https://github.com/potiuk> - Don't call get_connection from provide_bucket_name (#28716) <#28716>: @dstandish <https://github.com/dstandish> Provider apache.beam: 4.1.1rc1 <https://pypi.org/project/apache-airflow-providers-apache-beam/4.1.1rc1> - Ensure Beam Go file downloaded from GCS still exists when referenced (#28664) <#28664>: @johannaojeling <https://github.com/johannaojeling> Provider apache.hive: 5.1.1rc1 <https://pypi.org/project/apache-airflow-providers-apache-hive/5.1.1rc1> - Move local_infile option from extra to hook parameter (#28811) <#28811>: @potiuk <https://github.com/potiuk> Provider cncf.kubernetes: 5.1.1rc1 <https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/5.1.1rc1> - Fix Incorrect await_container_completion (#28771) <#28771>: @csm10495 <https://github.com/csm10495> Provider common.sql: 1.3.3rc1 <https://pypi.org/project/apache-airflow-providers-common-sql/1.3.3rc1> - Handle non-compliant behaviour of Exasol cursor (#28744) <#28744>: @potiuk <https://github.com/potiuk> Provider dbt.cloud: 2.3.1rc1 <https://pypi.org/project/apache-airflow-providers-dbt-cloud/2.3.1rc1> - Use entire tenant domain name in dbt Cloud connection (#28890) <#28890>: @josh-fell <https://github.com/josh-fell> Provider docker: 3.4.1rc1 <https://pypi.org/project/apache-airflow-providers-docker/3.4.1rc1> - Add correct widgets in Docker Hook (#28700) <#28700>: @potiuk <https://github.com/potiuk> - make docker operators always use DockerHook for API calls (#28363) <#28363>: @Taragolis <https://github.com/Taragolis> Provider elasticsearch: 4.3.3rc1 <https://pypi.org/project/apache-airflow-providers-elasticsearch/4.3.3rc1> - Allow nested attr in elasticsearch host_field (#28878) <#28878>: @dstandish <https://github.com/dstandish> Provider exasol: 4.1.3rc1 <https://pypi.org/project/apache-airflow-providers-exasol/4.1.3rc1> - Handle non-compliant behaviour of Exasol cursor (#28744) <#28744>: @potiuk <https://github.com/potiuk> Provider google: 8.8.0rc1 <https://pypi.org/project/apache-airflow-providers-google/8.8.0rc1> - Add deferrable GCSObjectExistenceSensorAsync (#28763) <#28763>: @rajaths010494 <https://github.com/rajaths010494> - Support partition_columns in BaseSQLToGCSOperator (#28677) <#28677>: @vchiapaikeo <https://github.com/vchiapaikeo> - BigQueryCreateExternalTableOperator fix field delimiter not working with csv (#28856) <#28856>: @fhaubner <https://github.com/fhaubner> - replace private _get_credentials by public get_credentials (#28588) <#28588>: @zazaho <https://github.com/zazaho> - FixGoogleCampaignManagerReportSensor with QUEUED status (#28735) <#28735>: @kacpermuda <https://github.com/kacpermuda> - Fix BigQueryColumnCheckOperator runtime error (#28796) <#28796>: @vchiapaikeo <https://github.com/vchiapaikeo> - Assign datasetReference by default in create_empty_dataset of BigQueryHook (#28782) <#28782>: @shahar1 <https://github.com/shahar1> Provider http: 4.1.1rc1 <https://pypi.org/project/apache-airflow-providers-http/4.1.1rc1> - Change logging for HttpHook to debug (#28911) <#28911>: @amoghrajesh <https://github.com/amoghrajesh> Provider jenkins: 3.2.0rc1 <https://pypi.org/project/apache-airflow-providers-jenkins/3.2.0rc1> - check jenkins result state (#28659) <#28659>: @tnyz <https://github.com/tnyz> Provider microsoft.azure: 5.1.0rc1 <https://pypi.org/project/apache-airflow-providers-microsoft-azure/5.1.0rc1> - Hook for managing directories and files in Azure Data Lake Storage Gen2 (#28262) <#28262>: @bharanidharan14 <https://github.com/bharanidharan14> - Hide extra field in Azure Blob Storage wasb (#28914) <#28914>: @IAL32 <https://github.com/IAL32> Provider microsoft.psrp: 2.2.0rc1 <https://pypi.org/project/apache-airflow-providers-microsoft-psrp/2.2.0rc1> - Add option to add arguments to PSRP hook and operator (#27689) <#27689>: @malthe <https://github.com/malthe> Provider mongo: 3.1.1rc1 <https://pypi.org/project/apache-airflow-providers-mongo/3.1.1rc1> - MongoHook optimization (#28675) <#28675>: @stamixthereal <https://github.com/stamixthereal> Provider mysql: 4.0.0rc1 <https://pypi.org/project/apache-airflow-providers-mysql/4.0.0rc1> - Move local_infile option from extra to hook parameter (#28811) <#28811>: @potiuk <https://github.com/potiuk> Provider slack: 7.2.0rc1 <https://pypi.org/project/apache-airflow-providers-slack/7.2.0rc1> - Add general-purpose "notifier" concept to DAGs (#28569) <#28569>: @ephraimbuddy <https://github.com/ephraimbuddy> Note: we decided to release Slack tough this specific feature is can be used only with Airflow 2.6 (see discussion <#28929 (comment)>) The guidelines on how to test providers can be found in Verify providers by contributors <https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors> All users involved in the PRs: @ShelRoman <https://github.com/ShelRoman> @stamixthereal <https://github.com/stamixthereal> @csm10495 <https://github.com/csm10495> @kacpermuda <https://github.com/kacpermuda> @Taragolis <https://github.com/Taragolis> @shahar1 <https://github.com/shahar1> @ferruzzi <https://github.com/ferruzzi> @vincbeck <https://github.com/vincbeck> @malthe <https://github.com/malthe> @josh-fell <https://github.com/josh-fell> @IAL32 <https://github.com/IAL32> @tlinkin <https://github.com/tlinkin> @vchiapaikeo <https://github.com/vchiapaikeo> @vandonr-amz <https://github.com/vandonr-amz> @bharanidharan14 <https://github.com/bharanidharan14> @tnyz <https://github.com/tnyz> @rajaths010494 <https://github.com/rajaths010494> @amoghrajesh <https://github.com/amoghrajesh> Committer - I acknowledge that I am a maintainer/committer of the Apache Airflow project. — Reply to this email directly, view it on GitHub <#28938>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAGOJKHGFQTNTCQE62DBBTWSKSLZANCNFSM6AAAAAAT3I2IKU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Taragolis]

• make docker operators always use DockerHook for API calls #28363: Test locally, work as expected
• Add correct widgets in Docker Hook #28700: Found that we labeled reauth as Disable SSL which is not correct, it is stand for refresh existing authentication on the Docker server, see: https://docker-py.readthedocs.io/en/stable/client.html#docker.client.DockerClient.login

[eladkal]

• Add correct widgets in Docker Hook #28700: Found that we labeled reauth as Disable SSL which is not correct, it is stand for refresh existing authentication on the Docker server, see: https://docker-py.readthedocs.io/en/stable/client.html#docker.client.DockerClient.login

can you open a PR with the fix? :)

[Taragolis]

can you open a PR with the fix? :)

Sure.

[csm10495]

#28771 still looks good.

[vandonr-amz]

my Sagemaker changes were covered by a system test that's still green, so good to go :)

[eladkal]

Docker provider will be excluded from this release and I will cut RC2 for it after this wave

[josh-fell]

#28890 Looks good!

[eladkal]

Thank you everyone.
Providers are released (except Docker which I will cut RC2 for)
I invite everyone to help improve providers for the next release, a list of open bugs can be found here.