Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(oio): Add MultipartUploadWrite to easier the work for Writer #2699

Merged
merged 3 commits into from
Jul 24, 2023

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Jul 24, 2023

This PR will add MultipartUploadWrite to easier the work of implementing Writer.

It's an internal API that users can't use directly. We add this to simplify the work to implement Writer for services like s3, cos and oss. And align the behavior of buffering and available options.

This PR just migrated the implement of s3 and we will start a migrate issue for other services.

Signed-off-by: Xuanwo <[email protected]>
Copy link
Member

@suyanhanx suyanhanx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM.

But for some services like Google Drive, they need position to indicate where to continue to write. We could count and record the sizes we uploaded, then we can also get the position.

core/src/raw/oio/write/multipart_upload_write.rs Outdated Show resolved Hide resolved
@Xuanwo
Copy link
Member Author

Xuanwo commented Jul 24, 2023

But for some services like Google Drive, they need position to indicate where to continue to write. We could count and record the sizes we uploaded, then we can also get the position.

This concern is invalid because only services such as S3 and OSS, which have a multipart upload API, can implement this feature. Other services like fs, google drive are not covered (maybe will have their own abstraction).

@suyanhanx
Copy link
Member

But for some services like Google Drive, they need position to indicate where to continue to write. We could count and record the sizes we uploaded, then we can also get the position.

This concern is invalid because only services such as S3 and OSS, which have a multipart upload API, can implement this feature. Other services like fs, google drive are not covered (maybe will have their own abstraction).

😯 Ok.

Copy link
Member

@suyanhanx suyanhanx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@Xuanwo Xuanwo merged commit a16dbec into main Jul 24, 2023
@Xuanwo Xuanwo deleted the multipart-writer branch July 24, 2023 08:21
@oowl oowl mentioned this pull request Jul 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants