Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(core): Split buffer logic from underlying storage operations #2903

Merged
merged 12 commits into from
Aug 23, 2023

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Aug 22, 2023

This PR splits buffer logic from underlying storage operations to make them more clear. This PR only migrates multipart uploads and append object for now. We can migrate gcs upload and azblob upload in next PRs.

Xuanwo added 4 commits August 22, 2023 15:21
Signed-off-by: Xuanwo <[email protected]>
Signed-off-by: Xuanwo <[email protected]>
Signed-off-by: Xuanwo <[email protected]>
Xuanwo added 2 commits August 22, 2023 17:25
Signed-off-by: Xuanwo <[email protected]>
Signed-off-by: Xuanwo <[email protected]>
@Xuanwo
Copy link
Member Author

Xuanwo commented Aug 22, 2023

  2023-08-22T09:37:51.848707Z ERROR opendal::services: service=s3 operation=Writer::close path=bc2a53e0-e316-4547-9761-00f1c4bc4297 written=51547533 -> data close failed: Unexpected (permanent) at Writer::close => S3Error { code: "InvalidPart", message: "All non-trailing parts must have the same length.", resource: "", request_id: "" }

Context:
   uri: ***/***/83d71cb1-c866-428c-b98d-b3524bd4f884/bc2a53e0-e316-4547-9761-00f1c4bc4297?uploadId=AJ1CTqPx%2Bfaq%2BlgT7QVTh%2BFIUs1948K2Q4I%2FqZJIVrYJx71WCHrMtXAzuC5bq07jYc4Q37Zx3vYkH69e5gIsqilzlmcPUwuqjzosxCJy1yBd09fsWFctpqcnbRLsmi3Iy7SRmkQ3IqeTsnlbXV0eKkPTGE2U6EMsD%2F2Lhpok7qZ%2BMUM84VgwunX2%2FpQpmSnXflykOfIQ9WV%2BOjhOv6%2F7ptB0NgQseVkVCLdreM8IPUJYIENWvPXkFhlZbEojnZ9ibrEWAmdADwWcGhhARG%2BdfhQbyVA%2Bj9eLWyL1RI2B%2F0n61OV4v%2BZoAC7L3PqmcOvRdOtsOP5wMNrdISEEo%2Be%2Btv4%3D
   response: Parts { status: 400, version: HTTP/1.1, headers: {"date": "Tue, 22 Aug 2023 09:37:51 GMT", "content-type": "application/xml", "content-length": "145", "connection": "keep-alive", "server": "cloudflare", "cf-ray": "7faa23b91daf3b12-IAD"} }
   service: s3
   path: bc2a53e0-e316-4547-9761-00f1c4bc4297

Special requirement of r2

@Xuanwo
Copy link
Member Author

Xuanwo commented Aug 23, 2023

cc @suyanhanx, please take another look, thanks!

Copy link
Member

@suyanhanx suyanhanx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@Xuanwo Xuanwo merged commit d740edb into main Aug 23, 2023
@Xuanwo Xuanwo deleted the buffer-refactor branch August 23, 2023 04:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants