fsspec base Hook for filesystem operations

[AlexisBRENON]

Related: #8804, #3526

As explained in the related issue, it can be hard to cover all the <provider>To<provider> use cases. It could be nice to have a generic "file transfer" operator which can handle most of the storage provider (local filesystem as well as cloud blob storages). Such kind of storage wrapper already exists as the fsspec library which is already an Airflow dependency.

This PR aims at providing a base "class" (it's more like a mixin) to mix with blob storage provider (such as S3, GCS, etc.) to allow easy creation of fsspec.FileSystem from such hooks.

I am not very fluent with the Hook/Connection API, so I am not sure that this PR make sense.

---

[potiuk]

We are about to unsuspend yandex provider (which was the main reason why the PR failed) . You should rebase after we merge it: #33574

To be honest, I am not sure if we need it. I think there is a very little benefit from having such "common" provider. We already have (and continue having) a big share of problems resulting from common.sql extraction - and there was a good reason for that extraction - i.e. Open Lineage integration might benefit from it.

With Filesstem, I am not sure what are the benefits. The key thing is -extracing code is cool (because of DRY) but it introduces coupling (i.e. all users of the common code become implicitly dependent). And it introduces all kinds of issues where change in the common code might lead to dubtle bugs in related providers.

I yet fail to see what would be the benefits of introducing such common code. But maybe I am short-sited?

[AlexisBRENON]

To be honest, I am not sure if we need it. I think there is a very little benefit from having such "common" provider. [...]
[...] it introduces coupling (i.e. all users of the common code become implicitly dependent). And it introduces all kinds of issues where change in the common code might lead to dubtle bugs in related providers.

I understand your point of view and agree that having such generic class mixed with a lot of other ones make changes hard to do and to predict all the impacts.

What about a less "common" provider, more focused on fsspec.Filesystem creation ?
My actual use case is to move some data from one bucket to another. Currently, my operator take two URIs as arguments, as well as one connection ID (for GCS access in my case). But this requires some rework if I need:

• to change one of the storage from GCS to S3 for exemple,
• different credentials for each buckets.

Having a FileSystem connection type would allow me to provision a connection for each of my storage service and using only two connections ID as arguments of the operator.

I yet fail to see what would be the benefits of introducing such common code. But maybe I am short-sited?

Do you think that this use case would benefit from introducing a FileSystem connection, or do I mis-conceived my operator and/or workflow ?

[potiuk]

To be honest, I am not sure if we need it. I think there is a very little benefit from having such "common" provider. [...]
[...] it introduces coupling (i.e. all users of the common code become implicitly dependent). And it introduces all kinds of issues where change in the common code might lead to dubtle bugs in related providers.

I understand your point of view and agree that having such generic class mixed with a lot of other ones make changes hard to do and to predict all the impacts.

What about a less "common" provider, more focused on fsspec.Filesystem creation ? My actual use case is to move some data from one bucket to another. Currently, my operator take two URIs as arguments, as well as one connection ID (for GCS access in my case). But this requires some rework if I need:

• to change one of the storage from GCS to S3 for exemple,
• different credentials for each buckets.

Having a FileSystem connection type would allow me to provision a connection for each of my storage service and using only two connections ID as arguments of the operator.

I yet fail to see what would be the benefits of introducing such common code. But maybe I am short-sited?

Do you think that this use case would benefit from introducing a FileSystem connection, or do I mis-conceived my operator and/or workflow ?

I suggest you to do a different thing. Write a taskflow operator for your case using hooks. I wrote a blog post about it some time ago - and IMHO using this kind of approach is way better than trying to make a "generic" transfer operator

https://medium.com/apache-airflow/generic-airflow-transfers-made-easy-5fe8e5e7d2c2

[Taragolis]

Just wondering, what kind of behaviour expected here?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.