Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

copyblocks: add support for S3-compatible buckets and object storage migration #5486

Merged
merged 9 commits into from
Jul 20, 2023

Conversation

andyasp
Copy link
Contributor

@andyasp andyasp commented Jul 12, 2023

What this PR does

copyblocks currently supports GCS->GCS and ABS->ABS server-side copies. This change adds:

  • S3->S3 server-side copy support
  • client-side copy support for copies to be performed across services (source GET chained to destination PUT)
  • the ability to toggle between server-side copies and client-side copies if both buckets are on the same service

so the new support looks like {ABS, GCS, S3} -> {ABS, GCS, S3}.

Notably --service was replaced with --source-service and --destination-service, since the services may not be the same.

I have not finished testing these changes against real buckets yet, so I'm opening this as a draft.

Tests Performed:

  • S3->GCS
  • GCS->S3
  • S3->ABS
  • ABS->S3
  • S3->S3 (client-side)
  • S3->S3 (server-side, tested locally with minio due to permissions)

The goal of this testing was line coverage by having each service be both a source and a destination in client-side copies and to exercise that the S3 client expects to be passed a correct content length during Upload.

Which issue(s) this PR fixes or relates to

Fixes: N/A

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@andyasp andyasp added the enhancement New feature or request label Jul 12, 2023
@andyasp andyasp force-pushed the aasp/copyblocks-s3-and-migration branch 2 times, most recently from 315a6ed to 3471a88 Compare July 12, 2023 14:34
@andyasp andyasp force-pushed the aasp/copyblocks-s3-and-migration branch from 3471a88 to fe0091d Compare July 12, 2023 15:19
@andyasp andyasp marked this pull request as ready for review July 18, 2023 20:28
@andyasp andyasp requested a review from a team as a code owner July 18, 2023 20:28
Copy link
Contributor

@charleskorn charleskorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, thanks @andyasp!

@andyasp
Copy link
Contributor Author

andyasp commented Jul 19, 2023

Thanks for keeping me honest with message consistency @charleskorn, those nits were helpful.

Copy link
Contributor

@charleskorn charleskorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing my feedback @andyasp!

@charleskorn charleskorn merged commit 2d4176d into main Jul 20, 2023
@charleskorn charleskorn deleted the aasp/copyblocks-s3-and-migration branch July 20, 2023 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants