Add GCS Requester Pays bucket support to GCSToS3Operator #32760

hankehly · 2023-07-22T06:35:14Z

Summary

This PR adds an optional user_project argument to GCSToS3Operator in order to support operations on Google Cloud Storage buckets that have the Requester Pays feature enabled.

Classes and methods affected:

GCSToS3Operator
GCSDeleteBucketOperator (change required for system test)
GCSHook.download
GCSHook.provide_file
GCSHook.provide_file_and_upload
GCSHook.upload
GCSHook.delete_bucket (change required for system test)
GCSHook.list

Testing

All affected classes and methods are covered in the below system test.

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_DEFAULT_REGION=
export GOOGLE_APPLICATION_CREDENTIALS=
export GCP_PROJECT_ID=
pytest --system amazon tests/system/providers/amazon/aws/example_gcs_to_s3.py

hankehly · 2023-07-22T06:36:45Z

airflow/providers/amazon/aws/transfers/gcs_to_s3.py


    def execute(self, context: Context) -> list[str]:
        # list all files in an Google Cloud Storage bucket
-        hook = GCSHook(
+        gcs_hook = GCSHook(


Changed names of "hook" and "files" variables to "gcs_hook" and "gcs_files" respectively to differentiate from s3 hook and files.

hankehly · 2023-07-22T06:40:16Z

airflow/providers/google/cloud/hooks/gcs.py

@@ -506,7 +529,7 @@ def _call_with_retry(f: Callable[[], None]) -> None:
                    continue

        client = self.get_conn()
-        bucket = client.bucket(bucket_name)
+        bucket = client.bucket(bucket_name, user_project=user_project)


You can add support for "requester pays" buckets by specifying user_project when creating a bucket object. All updates in this PR are basically to pass user_project from the operator level to this bucket object factory method.

hankehly · 2023-07-22T06:40:52Z

airflow/providers/google/cloud/hooks/gcs.py

@@ -639,7 +660,6 @@ def is_updated_before(self, bucket_name: str, object_name: str, ts: datetime) ->
        """
        blob_update_time = self.get_blob_update_time(bucket_name, object_name)
        if blob_update_time is not None:
-


Black formatter removed some superfluous space.

hankehly · 2023-07-22T06:44:45Z

tests/system/providers/amazon/aws/example_gcs_to_s3.py

+    create_gcs_bucket = GCSCreateBucketOperator(
+        task_id="create_gcs_bucket",
+        bucket_name=gcs_bucket,
+        resource={"billing": {"requesterPays": True}},


To test that GCSToS3Operator works with requester pays buckets, you need a bucket with "requester pays" enabled. Luckily, you don't need two GCP accounts to use this feature. You can specify your own GCP project ID as the billable project. If a bucket has "requester pays" enabled and you don't specify a user_project, an error will occur.

hankehly · 2023-07-22T06:52:43Z

Static checks passing locally

hankehly · 2023-07-22T08:35:58Z

@eladkal @o-nikolas Please review this at your earliest convenience!

vincbeck · 2023-07-27T14:16:27Z

tests/providers/google/cloud/hooks/test_gcs.py

@@ -503,7 +503,7 @@ def test_delete_bucket(self, mock_service):

        self.gcs_hook.delete_bucket(bucket_name=test_bucket)

-        mock_service.return_value.bucket.assert_called_once_with(test_bucket)
+        mock_service.return_value.bucket.assert_called_once_with(test_bucket, user_project=None)


Is this necessary? It is the default value

Yeah I thought it was a little silly, but the assertion method recognizes the difference between specifying None upfront and leaving it as the default value. I only updated this code because CI tests were failing.

ferruzzi · 2023-07-27T19:17:06Z

tests/system/providers/amazon/aws/example_gcs_to_s3.py

    # [START howto_transfer_gcs_to_s3]
    gcs_to_s3 = GCSToS3Operator(
        task_id="gcs_to_s3",
-        bucket=s3_bucket,
-        dest_s3_key=s3_key,
+        bucket=gcs_bucket,


I can't believe that's been there this whole time. Maybe renaming that param to gcs_bucket should be on the todo list

That's a good idea. I'd be happy to change the declaration (keeping support for the old param name and adding a deprecation warning) in another PR.

Oh yeah, I definitely didn't mean to add to the scope of this one. If you want to take it in another PR, that's great. Ping me in it and I'll review it for you as well.

The AWS team runs "all" of the system tests regularly with a dashboard and fixes any regressions, but doesn't run any of the ones that require outside services (like maintaining GCS or MongoDB accounts) or that would have been caught long ago. [Speaking for myself here and not on behalf of anyone else] It would be neat to see if the Google team and the AWS team could work something out to get that combination of tests running somehow.

hankehly added 2 commits July 22, 2023 15:20

Add requester pays bucket support to GCSToS3Operator

6a8aa3c

Update docstrings

8f86014

boring-cyborg bot added area:providers area:system-tests provider:amazon AWS/Amazon - related issues provider:google Google (including GCP) related issues labels Jul 22, 2023

hankehly commented Jul 22, 2023

View reviewed changes

isort

46e019f

hankehly commented Jul 22, 2023

View reviewed changes

hankehly marked this pull request as ready for review July 22, 2023 06:52

hankehly requested review from eladkal and o-nikolas as code owners July 22, 2023 06:52

hankehly added 2 commits July 22, 2023 16:15

Fix failing unit tests

641ee0e

Fix failing test

82458a1

eladkal requested review from ferruzzi and vincbeck July 27, 2023 12:18

vincbeck reviewed Jul 27, 2023

View reviewed changes

ferruzzi reviewed Jul 27, 2023

View reviewed changes

hankehly requested review from vincbeck and ferruzzi July 27, 2023 23:22

moiseenkov approved these changes Jul 28, 2023

View reviewed changes

vincbeck approved these changes Jul 31, 2023

View reviewed changes

ferruzzi merged commit 915f9e4 into apache:main Jul 31, 2023

hankehly deleted the feature/issue-31137-gcs-to-s3 branch July 31, 2023 21:57

hankehly mentioned this pull request Aug 2, 2023

Rename "bucket" to "gcs_bucket" in GCSToS3Operator #33031

Merged

eladkal mentioned this pull request Aug 4, 2023

Status of testing Providers that were prepared on August 05, 2023 #33132

Closed

44 tasks

This was referenced Aug 8, 2023

Status of testing Providers that were prepared on August 08, 2023 #33193

Closed

Status of testing Providers that were prepared on August 09, 2023 #33257

Closed

eladkal mentioned this pull request Aug 23, 2023

GCSHook's functions for list and download do not work with Requester Pays buckets. #31137

Closed

2 tasks

kylase mentioned this pull request Feb 14, 2025

Fix exists method to support using Requester Pays #46759

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GCS Requester Pays bucket support to GCSToS3Operator #32760

Add GCS Requester Pays bucket support to GCSToS3Operator #32760

hankehly commented Jul 22, 2023 •

edited

Loading

hankehly Jul 22, 2023

hankehly Jul 22, 2023

hankehly Jul 22, 2023

hankehly Jul 22, 2023

hankehly commented Jul 22, 2023

hankehly commented Jul 22, 2023

vincbeck Jul 27, 2023

hankehly Jul 27, 2023

ferruzzi Jul 27, 2023

hankehly Jul 27, 2023

ferruzzi Jul 28, 2023

Add GCS Requester Pays bucket support to GCSToS3Operator #32760

Add GCS Requester Pays bucket support to GCSToS3Operator #32760

Conversation

hankehly commented Jul 22, 2023 • edited Loading

Summary

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hankehly commented Jul 22, 2023

hankehly commented Jul 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hankehly commented Jul 22, 2023 •

edited

Loading