kvclient: introduce txn interceptor to buffer writes #139053
Assignees
Labels
A-buffered-writes
Related to the introduction of buffered writes
A-kv-transactions
Relating to MVCC and the transactional model.
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-kv
KV Team
Comments
arulajmani
added
A-buffered-writes
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 24, 2025
This patch introduces a new txn interceptor to the TxnCoordSender's interceptor stack -- the txnWriteBuffer. This commit is mostly boilerplate code with some forward looking commentary, so for now, the interceptor is as no-op as it gets. Soon, we'll expand this to actually introduce client side write buffering. References cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 24, 2025
This patch introduces a new txn interceptor to the TxnCoordSender's interceptor stack -- the txnWriteBuffer. This commit is mostly boilerplate code with some forward looking commentary, so for now, the interceptor is as no-op as it gets. Soon, we'll expand this to actually introduce client side write buffering. References cockroachdb#139053 Release note: None
craig bot
pushed a commit
that referenced
this issue
Jan 24, 2025
139155: cdctest: add mvcc_timestamp, diff, sink_config options to nemesis testing r=asg0451 a=aerfrei This work adds to random changefeed settings we pass into changefeeds in nemesis testing. This commit adds support for: changefeeds without the diff option specified, the mvcc_timestamp option and the kafka and pubsub sink configs. See also: #134119 Epic: [CRDB-42866](https://cockroachlabs.atlassian.net/browse/CRDB-42866) Release note: None 139377: sqlstats: remove unnecessary interfaces r=xinhaoz a=dhartunian The `Reader`, `Writer`, `ApplicationStats`, `Storage`, and `Provider` interfaces were overcomplicating the implementation and making the code harder to navigate and understand. In practice, there are generally single implementations of these interfaces so it's much simpler to just use the concrete struct pointers. Epic: CRDB-45771 Release note: None 139649: kvfeed: fix span frontier minimum timestamp r=andyyang890 a=wenyihu6 When calculating the minimum timestamp of spans in the frontier, we have a similar issue to #123371. Specifically, if the minimal timestamp tracked so far is empty, we always take the span's timestamp as the minimal timestamp. Release note: none Epic: none 139738: cli: update maybeWarnMemorySizes to use redact.StringBuilder r=rafiss a=rafiss Using a redact.StringBuilder is a bit safer than using a SafeString cast on the whole message, since if we ever add sensitive values to this message in the future, it will automatically take care of redacting them. follow up to #139117 Epic: CRDB-37533 Release note: None 139742: kvclient: introduce txnWriteBuffer interceptor r=stevendanna a=arulajmani This patch introduces a new txn interceptor to the TxnCoordSender's interceptor stack -- the txnWriteBuffer. This commit is mostly boilerplate code with some forward looking commentary, so for now, the interceptor is as no-op as it gets. Soon, we'll expand this to actually introduce client side write buffering. References #139053 Release note: None Co-authored-by: Aerin Freilich <[email protected]> Co-authored-by: David Hartunian <[email protected]> Co-authored-by: Wenyi Hu <[email protected]> Co-authored-by: Rafi Shamim <[email protected]> Co-authored-by: Arul Ajmani <[email protected]>
andy-kimball
pushed a commit
to andy-kimball/cockroach
that referenced
this issue
Jan 27, 2025
This patch introduces a new txn interceptor to the TxnCoordSender's interceptor stack -- the txnWriteBuffer. This commit is mostly boilerplate code with some forward looking commentary, so for now, the interceptor is as no-op as it gets. Soon, we'll expand this to actually introduce client side write buffering. References cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 28, 2025
This patch introduces a btree to buffer writes on the txnWriteBuffer interceptor. References cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 28, 2025
This patch introduces a btree to buffer writes on the txnWriteBuffer interceptor. References cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 29, 2025
This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separte read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 29, 2025
This patch introduces a btree to buffer writes on the txnWriteBuffer interceptor. References cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 29, 2025
This patch introduces a btree to buffer writes on the txnWriteBuffer interceptor. References cockroachdb#139053 Release note: None
craig bot
pushed a commit
that referenced
this issue
Jan 29, 2025
138717: settings: increase number of cores test can use r=cthumuluru-crdb a=cthumuluru-crdb Clutser setting updates on one node occasionally take over 45 seconds to propagate to other nodes. This test has failed a few times recently. The issue is not reproducible, even under stress testing with 10,000 repetitions. Increasing the timeout to 60 seconds to determine if it resolves the problem. Epic: None Fixes: #133732 Release note: None 139958: cdctest: fix fingerprint validator primary key cols check r=asg0451 a=wenyihu6 Previously, fingerprint validator could fail with sql smith due to precision loss from JSON encoding & decoding of large numbers. We previously fixed the validator regarding its values check. This patch fixes the primary key columns check accordingly. Part of: #124146 Release note: none 139963: kvclient: add a btree to store buffered writes r=stevendanna a=arulajmani This patch introduces a btree to buffer writes on the txnWriteBuffer interceptor. References #139053 Release note: None Co-authored-by: Chandra Thumuluru <[email protected]> Co-authored-by: Wenyi Hu <[email protected]> Co-authored-by: Arul Ajmani <[email protected]>
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 5, 2025
This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separte read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 5, 2025
This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separte read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 5, 2025
This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separte read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 5, 2025
This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separte read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 5, 2025
This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separte read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 6, 2025
This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separate read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 6, 2025
This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separate read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes cockroachdb#139053 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 7, 2025
This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separate read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes cockroachdb#139053 Release note: None
craig bot
pushed a commit
that referenced
this issue
Feb 7, 2025
140001: kvclient: buffer blind writes on the client r=stevendanna,yuzefovich a=arulajmani First commit from #139963. ---- This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separte read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes #139053 Release note: None Co-authored-by: Arul Ajmani <[email protected]>
craig bot
pushed a commit
that referenced
this issue
Feb 7, 2025
140001: kvclient: buffer blind writes on the client r=stevendanna,yuzefovich a=arulajmani This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separte read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes #139053 Release note: None 140066: server: do not use MustBeDString on nullable result col r=xinhaoz a=xinhaoz In the data distribution handler we were attempting to read a `raw_sql_config` on `crdb_internal.zones` using `MustBeDString` which panics if the value is null. This column is nullable. We now allow null values to be read and make the response value an empty string in that case. Fixes: #140044 Release note (bug fix): Data distribution page in advanced debug will no longer crash if there are null values for `raw_sql_config` in `crdb_internal.zones`. Co-authored-by: Arul Ajmani <[email protected]> Co-authored-by: Xin Hao Zhang <[email protected]>
craig bot
pushed a commit
that referenced
this issue
Feb 7, 2025
140001: kvclient: buffer blind writes on the client r=stevendanna,yuzefovich a=arulajmani This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separte read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes #139053 Release note: None Co-authored-by: Arul Ajmani <[email protected]>
craig bot
pushed a commit
that referenced
this issue
Feb 7, 2025
140001: kvclient: buffer blind writes on the client r=stevendanna,yuzefovich a=arulajmani This patch adds logic to buffer blind writes (Puts or Deletes) in the txnWriteBuffer. Interceptors and layers above remain oblivious to the txnWriteBuffer's decision to buffer any writes. All buffered writes are flushed at commit time, in the same batch as the EndTxn request. This flushing at commit time is also hidden from interceptors above the txnWriteBuffer by stripping responses on the return path. The code structure here allows us to split a batch request into different bits, where some portion is evaluated locally and the rest is sent to the KV layer. It's written with a future where we split read-write requests (e.g. CPuts) into separte read/write halves, where the read portion needs to be evaluated at the leaseholder, but the write needs to be buffered. Closes #139053 Release note: None Co-authored-by: Arul Ajmani <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In service of #72614
We'll want to create a new
txnInterceptorfor theTxnCoordSender's interceptor stack to buffer writes that will be flushed at commit time.Jira issue: CRDB-46539
The text was updated successfully, but these errors were encountered: