Moving yandex provider to new provider structure

[kunaljubce]

Relates to #46045, moving yandex provider to new folder structure. Also added a check in the move_providers.py script to check if a dir exists before trying to move it.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

OK. there is a small caveat with "latest botocore" - Let me add a fixup - I know how to fix it

[kunaljubce]

When I try to reproduce this error locally with breeze testing core-tests --run-in-parallel, it returns errors but seemingly unrelated to yandex provider changes -

=========================== short test summary info ============================
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/ml_engine/example_mlengine.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/ml_engine/example_mlengine.py': 'Traceback (most recent call last):\n  File "<frozen importlib._bootstrap_external>", line 1040, in get_data\n  File "/opt/airflow/airflow/utils/timeout.py", line 69, in handle_timeout\n    raise AirflowTaskTimeout(self.error_message)\nairflow.exceptions.AirflowTaskTimeout: DagBag import timeout for /opt/airflow/providers/google/tests/system/google/cloud/ml_engine/example_mlengine.py after 30.0s.\nPlease take a look at these docs to improve your DAG import time:\n* http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/best-practices.html#top-level-python-code\n* http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/best-practices.html#reducing-dag-complexity, PID: 75\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/ml_engine/example_mlengine.py': 'Traceback (most recent call...-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/best-practices.html#reducing-dag-complexity, PID: 75\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/ml_engine/example_mlengine.py': 'Traceback (most recent call...-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/best-practices.html#reducing-dag-complexity, PID: 75\n'} = <airflow.models.dagbag.DagBag object at 0xffff48087910>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_forecasting_training.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_forecasting_training.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_forecasting_training.py'...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_forecasting_training.py'...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43775190>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_object_detection.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_object_detection.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_object_detection.p...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_object_detection.p...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff434f20a0>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_training.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_training.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_training.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_training.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43838430>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_list_training.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_list_training.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_list_training.py': 'Trac...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_list_training.py': 'Trac...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff4341ac40>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_tabular_training.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_tabular_training.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_tabular_training.py': 'T...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_tabular_training.py': 'T...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff434cab50>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_tracking.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_tracking.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_tracking.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_tracking.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff4383ec40>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_training.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_training.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_training.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_training.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43755f40>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_batch_prediction_job.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_batch_prediction_job.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_batch_prediction_job.py': 'Trace...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_batch_prediction_job.py': 'Trace...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43880880>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_container.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_container.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_container.py': 'Traceback...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_container.py': 'Traceback...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff4389ae20>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job.py': 'Traceback (most...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job.py': 'Traceback (most...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43549b20>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job_python_package.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job_python_package.py': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n    from google.cloud.aiplatform_v1 import gapic_version as package_version\n  File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n    from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
 +  where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job_python_package.py': '...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
 +    where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job_python_package.py': '...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43955cd0>.import_errors
= 12 failed, 1549 passed, 1 skipped, 1 xfailed, 1 warning in 1577.55s (0:26:17) =
No stopped containers

There were errors when running some tasks. Quitting.

[potiuk]

pushed likely fix

[kunaljubce]

Thanks @potiuk. For my understanding, can you please explain a little about the botocore issue?

[potiuk]

Thanks @potiuk. For my understanding, can you please explain a little about the botocore issue?

Yeah. The thing is that with botocore tests we are upgrading to latest (i.e. newest) possible botocore - basically by removing aiobotocore that holds it back. The aiobotocore is non-aws, 3rd-party async botocore extension and the problem with it is that it lags behind the main botocore - which is released almost daily

See: https://pypi.org/project/botocore/#history
vs. https://pypi.org/project/aiobotocore/#history

botocore - released today, aiobotocore released 22 Jan and it has Requires-Dist: botocore<1.36.4,>=1.36.0 -> 1.36.0 of botocore was released on 15th of Jan.

So those tests are removing aiobotocore and try to run all the tests with latest botocore. And yandex also uses a aiobotocore I think.

I see my fix did not work, so I have to fix it differently

[potiuk]

Ok. I pushed simplified version. I think in this case we do not have to run pip check - because this is what caused the failures. When we simply upgrade botocore to latest version it will cause conflicting dependencies with other package(s) - - but this should be no big issue , as long as all the tests pass.

[potiuk]

🤞

[jscheffl]

Are you sure you want to delete all the RST when moving the provider?

[potiuk]

Are you sure you want to delete all the RST when moving the provider?

AAAAAH... Merged tooo fast... good eye Jens.

[potiuk]

Ah no --... yandex docs were copied previously (accidentally) as part of another provider (TERADATA) :) . All good :)

[jscheffl]

Ah no --... yandex docs were copied previously (accidentally) as part of another provider (TERADATA) :) . All good :)

Did not know but good you confirmed.... and no Aaaah :-D