Fix for idempotent producer fatal errors, triggered after a possibly persisted message state #4438
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
emasab
force-pushed
the
dev_fix_idempotent_producer_fatal_errors
branch
from
547e28b to
8f36dc5
Compare
pranavrth
approved these changes
Sep 29, 2023
axelandersson
added a commit
to axelandersson/librdkafka
that referenced
this pull request
Oct 5, 2023
* upstream/master: librdkafka v2.3.0 (confluentinc#4455) Fix for idempotent producer fatal errors, triggered after a possibly persisted message state (confluentinc#4438) Move can_q_contain_fetched_msgs inside q_serve (confluentinc#4431) [KIP-580] Exponential Backoff with Mock Broker Changes to Automate Testing. (confluentinc#4422) Update only the mklove version of OpenSSL to 3.0.11 (confluentinc#4454) Permanent errors during offset validation should be retried (confluentinc#4447) Increased flexver request size for Metadata request to include topic_id size (confluentinc#4453) Fix loop of OffsetForLeaderEpoch requests on quick leader changes (confluentinc#4433) Fix for stored offsets not being committed if they lacked the leader epoch (confluentinc#4442) Add leader epoch to control messages (confluentinc#4434) Refactored tmpabuf and fixed an insufficient buffer allocation (confluentinc#4449) Work around KIP-700 restrictions for DescribeCluster [KIP-430] [admin] KIP-430: Add authorized operations to describe API Fix segfault if assignor state is NULL, (confluentinc#4381)
8 tasks
airlock-confluentinc bot
pushed a commit
that referenced
this pull request
Nov 26, 2024
…persisted message state (#4438)
airlock-confluentinc bot
pushed a commit
that referenced
this pull request
Nov 26, 2024
…persisted message state (#4438)
Merged
airlock-confluentinc bot
pushed a commit
that referenced
this pull request
Nov 26, 2024
…persisted message state (#4438)
anchitj
added a commit
that referenced
this pull request
Dec 15, 2024
* Fix for idempotent producer fatal errors, triggered after a possibly persisted message state (#4438) * Remove CentOS 6 and CentOS 7 binaries (#4775) *Remove CentOS 6 and 7 support as discontinued, keeps using manylinux_2_28 based on AlmaLinux 8 (CentOS 8) * Remove fix for CentOS 6 * Add CHANGELOG entry * Upgrade test and verify package creation or installation using clients repository * Upgrade msvcr140 and vcpkg dependencies (#4872) * Add forward declaration to fix compilation without ssl (#4794) and add build checks with different configurations * PR comments * Add files for lz4 1.9.4 (#4726) * Add files for lz4 1.9.4 * Update changelog.md * rdxxhash should not be in clang-format list * Add instructions and update memory alloc/free * Update instructions for lz4 * NONJAVACLI-3460: update dependencies (#4706) * update third party dependencies * update lz4 version in the header file * update libraries for the windows build * reverting the version bump in the headers * use the latest version of curl * Update OpenSSL and add CHANGELOG.md * downgade curl version to one available via vcpkg * downgrade zlib to last available version in vcpkg * downgrade zstd to the latest available * Include CPPFLAGS within make for libcurl * Update mklove/modules/configure.libcurl * Update CHANGELOG.md --------- Co-authored-by: Milind L <[email protected]> Co-authored-by: Emanuele Sabellico <[email protected]> * Upgrade linux dependencies (#4875) * Security upgrade for OpenSSL and Curl, CVEs fixed: OpenSSL - CVE-2024-2511 - CVE-2024-4603 - CVE-2024-4741 - CVE-2024-5535 - CVE-2024-6119 CURL - CVE-2024-8096 - CVE-2024-7264 - CVE-2024-6874 - CVE-2024-6197 * Fix for curl configure failure caused by curl/curl#14373 * Include NOTE in CHANGELOG * Update RD_KAFKA_VERSION in rdkafkacpp.h --------- Co-authored-by: Emanuele Sabellico <[email protected]> Co-authored-by: Milind L <[email protected]> Co-authored-by: Jan Werner <[email protected]> Co-authored-by: Milind L <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After a possibly persisted error, such as a disconnection or a timeout, next expected sequence was increased, leading to a fatal error if the message wasn't persisted and the second one in queue failed with an
OUT_OF_ORDER_SEQUENCE_NUMBER.The error could contain the message "sequence desynchronization" with just one possibly persisted error or "rewound sequence number" in case of multiple errored messages.
Solved by treating the possible persisted message as not persisted, and expecting a
DUPLICATE_SEQUENCE_NUMBERerror in case it was orNO_ERRORin case it wasn't, in both cases the message will be considered delivered.