Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

object_store: delete multiple #4154

Closed
Kinrany opened this issue Apr 28, 2023 · 3 comments
Closed

object_store: delete multiple #4154

Kinrany opened this issue Apr 28, 2023 · 3 comments
Labels
development-process Related to development process of arrow-rs enhancement Any new improvement worthy of a entry in the changelog

Comments

@Kinrany
Copy link
Contributor

Kinrany commented Apr 28, 2023

Problem

I want to delete multiple objects atomically.

Solution

Add a new method to ObjectStore. It would follow the API similar to S3 DeleteObjects

Alternatives

Multiple calls to delete aren't atomic.

Discussion

Not all implementations of ObjectStore support this. We could have an extension trait, ObjectStoreAtomic: ObjectStore.

@Kinrany Kinrany added the enhancement Any new improvement worthy of a entry in the changelog label Apr 28, 2023
@Kinrany
Copy link
Contributor Author

Kinrany commented Apr 28, 2023

Would you consider accepting a PR that added this? Any thoughts on implementation?

@tustvold
Copy link
Contributor

tustvold commented Apr 28, 2023

I believe this is a duplicate of #2615

FWIW S3 DeleteObjects is not atomic, it can partially fail - https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html

In general S3 does not support any atomic operations, it does now provide read-after-write consistency, but it provides no atomic APIs. There is no CreateIfNotExists functionality for example

@Kinrany
Copy link
Contributor Author

Kinrany commented Apr 28, 2023

Agreed. And thanks, good to know that about S3.

@Kinrany Kinrany closed this as completed Apr 28, 2023
@tustvold tustvold added the development-process Related to development process of arrow-rs label Apr 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
development-process Related to development process of arrow-rs enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

No branches or pull requests

2 participants