Skip to content

Commit

Permalink
Revert "feat: transaction files and commit conflict resolution (#1127)"
Browse files Browse the repository at this point in the history
This reverts commit 0a12b97.
  • Loading branch information
eddyxu committed Aug 20, 2023
1 parent 0a84cd3 commit c560d81
Show file tree
Hide file tree
Showing 14 changed files with 165 additions and 1,355 deletions.
Binary file removed docs/_static/conflict_resolution_flow.png
Binary file not shown.
32 changes: 0 additions & 32 deletions docs/format.rst
Original file line number Diff line number Diff line change
Expand Up @@ -260,35 +260,3 @@ put-if-not-exists, these operations should be used. This is true of local file
systems and cloud object stores, with the notable except of AWS S3. For ones
that lack this functionality, an external locking mechanism can be configured
by the user.

Conflict resolution
~~~~~~~~~~~~~~~~~~~

If two writers try to commit at the same time, one will succeed and the other
will fail. The failed writer should attempt to retry the commit, but only if
it's changes are compatible with the changes made by the successful writer.

The changes for a given commit are recorded as a transaction file, under the
``_transactions`` prefix in the dataset directory. The transaction file is a
serialized ``Transaction`` protobuf message. See the ``transaction.proto`` file
for its definition.

.. image:: _static/conflict_resolution_flow.png

The commit process is as follows:

1. The writer finishes writing all data files.
2. The writer creates a transaction file in the ``_transactions`` directory.
This files describes the operations that were performed, which is used for two
purposes: (1) to detect conflicts, and (2) to re-build the manifest during
retries.
3. Look for any new commits since the writer started writing. If there are any,
read their transaction files and check for conflicts. If there are any
conflicts, abort the commit. Otherwise, continue.
4. Build a manifest and attempt to commit it to the next version. If the commit
fails because another writer has already committed, go back to step 3.
5. If the commit succeeds, update the ``_latest.manifest`` file.

When checking whether two transactions conflict, be conservative. If the
transaction file is missing, assume it conflicts. If the transaction file
has an unknown operation, assume it conflicts.
12 changes: 0 additions & 12 deletions protos/format.proto
Original file line number Diff line number Diff line change
Expand Up @@ -100,18 +100,6 @@ message Manifest {
//
// For a single file, will be zero.
uint32 max_fragment_id = 11;

// Path to the transaction file, relative to `{root}/_transactions`
//
// This contains a serialized Transaction message representing the transaction
// that created this version.
//
// May be empty if no transaction file was written.
//
// The path format is "{read_version}-{uuid}.txn" where {read_version} is the
// version of the table the transaction read from, and {uuid} is a
// hyphen-separated UUID.
string transaction_file = 12;
} // Manifest

// Auxiliary Data attached to a version.
Expand Down
119 changes: 0 additions & 119 deletions protos/transaction.proto

This file was deleted.

6 changes: 1 addition & 5 deletions rust/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,7 @@ fn main() -> Result<()> {
let mut prost_build = prost_build::Config::new();
prost_build.protoc_arg("--experimental_allow_proto3_optional");
prost_build.compile_protos(
&[
"./protos/format.proto",
"./protos/index.proto",
"./protos/transaction.proto",
],
&["./protos/format.proto", "./protos/index.proto"],
&["./protos"],
)?;

Expand Down
Loading

0 comments on commit c560d81

Please sign in to comment.