Support bucket with different priorities

[simolus3]

No description provided.

[simolus3]

Adding some of my own comments for a discussion on this.

[rkistner]

After checking the implementation, I realized that a row present in multiple buckets with different priorities has a big potential for edge cases - both in the spec and the specific implementation. This is specifically relevant for the r.bucket IN (SELECT id FROM involved_buckets) clause for example.

A potential use case could be where a user has a lot of data, and "stars" specific items to prioritize the sync of them.

For these examples, suppose we have two buckets: bucket1 and bucket2, with priorities 1 and 2 respectively. The same row could be in either or both of the buckets.

Current state

Case 1: A row is present in bucket 2, then added to bucket1. We get a partial_checkpoint_complete for bucket1.
The row will be included here, getting the latest version. ✅

Case 2: A row is removed from bucket2, and added to bucket1 at the same time ("starring" a row to move it to a higher priority). We get a partial_checkpoint_complete for bucket1. The row will be included here, and whether or not we got the REMOVE on bucket2 already doesn't make a difference. When we sync the rest of the checkpoint, the updated row will stay present. ✅

Case 3: A row is removed from bucket1, and added to bucket2 at the same time (removing a star to move to lower priority). We get a partial_checkpoint_complete for bucket1. We don't track removes per bucket, so this does nothing. ✅ When we sync the rest of the checkpoint, the row will be added again. ✅

Case 4: A row is present in bucket1 and bucket2, then removed from bucket1. We get a partial_checkpoint_complete for bucket1. We don't track removes per bucket, so this does nothing. ✅ When we sync the rest of the checkpoint, the row is updated with the state from bucket2. ✅

Hypothetical, if we tracked remove operations per bucket

Case 1: A row is present in bucket 2, then added to bucket1. We get a partial_checkpoint_complete for bucket1.
The row will be included here, getting the latest version. ✅

Case 2: A row is removed from bucket2, and added to bucket1 at the same time ("starring" a row to move it to a higher priority). We get a partial_checkpoint_complete for bucket1. The row will be included here, and whether or not we got the REMOVE on bucket2 already doesn't make a difference. When we sync the rest of the checkpoint, the updated row will stay present. ✅

Case 3: A row is removed from bucket1, and added to bucket2 at the same time (removing a star to move to lower priority). We get a partial_checkpoint_complete for bucket1. The row will be removed here, whether or not we got the PUT for bucket2. This works "according to the spec", but not sure if this is desired behavior. ❓ When we sync the rest of the checkpoint, the row will be added again. ✅

Case 4: A row is present in bucket1 and bucket2, then removed from bucket1. We get a partial_checkpoint_complete for bucket1. The row will be removed here, despite it still being present in bucket2. ❓ When we sync the rest of the checkpoint, the row will not be added again, resulting in an inconsistency. ❌

My summary from this that adding support for REMOVE operations in partial checkpoints does not actually give improved consistency as I hoped - it just creates more weird edge cases. The current behavior of only applying the REMOVE operations in the final checkpoint gives better results.