Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use transactions in commit API #1091

Closed
wjones127 opened this issue Jul 24, 2023 · 3 comments
Closed

Use transactions in commit API #1091

wjones127 opened this issue Jul 24, 2023 · 3 comments
Assignees
Labels
rust Rust related tasks

Comments

@wjones127
Copy link
Contributor

Right now, to commit a new version of a dataset the client must construct a Manifest correctly. This is a bit dangerous as serious issues could be caused by not constructing this correctly. In addition, manifests aren't easy to use to do conflict resolution on transactions.

We should switch to an API around transactions, that allow committing a transaction to a dataset, rather than a completely constructed state.

There could be a struct:

pub struct Transaction {
    updated_metadata: DatasetMetadata,
    new_fragments: Vec<Fragment>,
    updated_fragments: Vec<Fragment>,
    deleted_fragment_ids: Vec<u32>,
}

Then to commit a transaction, you could call either:

impl Dataset {
    /// Commit a transaction to an existing table
    pub async fn commit(
        &mut self,
        transaction: Transaction,
        retry_config: RetryConfig,
    ) -> Result<()> { ... }

    /// Commit a new table
    pub async fn commit_new(transaction: Transaction) -> Result<Self> { ... }
}

These methods could perform validation to make sure the transaction is correct, and can construct a valid manifest based on the state of the table. If a concurrent transaction beats the commit, then it could attempt to rebuild the manifest to create the next transaction, if that is allowed by conflict resolution algorithms.

@wjones127 wjones127 added the rust Rust related tasks label Jul 24, 2023
@wjones127
Copy link
Contributor Author

We added transaction concept in #1127, but we need to add them to the commit API.

@wjones127 wjones127 self-assigned this Aug 22, 2023
@wjones127 wjones127 changed the title Add transaction abstraction to help with commits Use transactions in commit API Aug 28, 2023
@wjones127
Copy link
Contributor Author

Note: someone started this in #982, so we should let them know when we implement this.

@wjones127
Copy link
Contributor Author

Completed in #1194.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
rust Rust related tasks
Projects
None yet
Development

No branches or pull requests

2 participants