changefeedccl: allow user to configure kafka server version #65389
Conversation
|
This change is |
|
Confluent provides docker containers with versions back to 3.0. It might be nice to have some nightly tests that uses those containers to test against a variety of versions. |
stevendanna
force-pushed
the
ssd/configurable-kafka-version
branch
from
09f3812 to
61e3c64
Compare
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @HonoreDB and @stevendanna)
pkg/ccl/changefeedccl/sink_kafka.go, line 441 at r1 (raw file):
*k = kafkaVersion(parsedVersion) return nil }
You could probably drop this custom parsing struct if store Version as a plain string in saramaConfig and change Apply signature to return an error.
There was a problem hiding this comment.
Reviewable status:
pkg/ccl/changefeedccl/sink_kafka.go, line 441 at r1 (raw file):
Previously, miretskiy (Yevgeniy Miretskiy) wrote…
You could probably drop this custom parsing struct if store Version as a plain string in saramaConfig and change Apply signature to return an error.
😆 I agree, that is what I originally had and then changed it to this when I saw the json parsing struct.
stevendanna
force-pushed
the
ssd/configurable-kafka-version
branch
from
61e3c64 to
0709891
Compare
The upgrade to the sarama library in facbd60 included a change to the default Version assumed by sarama from 0.8.2.0 to 1.0.0. The result of this is that we no longer support Kafka < 1.0 (Confluent Platform < 4.0) out of the box. All of the impacted versions are EOL according to: https://docs.confluent.io/platform/current/installation/versions-interoperability.html However, it may be nice to allow users still on those versions to continue to use changefeeds. While we could reduce the Version constant back to its original value, that comes at the cost of potentially making the Kafka server do more work to handle the older protocol messages. This change allows the user to set the version via the kafka_sink_config option. For example, users who need to connect to Confluent 3 would use this new option like so: ``` CREATE CHANGEFEED FOR target_table INTO 'kafka://localhost:9092' WITH kafka_sink_config='{"version": "0.8.2.0"}'; ``` Release note (enterprise): `kafka_sink_config` now supports a `version` configuration item to specify Kafka server versions. This is likely only necessary for old (Kafka 0.11/Confluent 3.3 or early) Kafka servers. Further, unspecified items in `kafka_sink_config` now retain their default values.
stevendanna
force-pushed
the
ssd/configurable-kafka-version
branch
from
0709891 to
d207058
Compare
|
bors r=miretskiy |
|
Build succeeded: |
The upgrade to the sarama library in
facbd60 included a change to the
default Version assumed by sarama from 0.8.2.0 to 1.0.0.
The result of this is that we no longer support Kafka < 1.0 (Confluent
Platform < 4.0) out of the box.
All of the impacted versions are EOL according to:
https://docs.confluent.io/platform/current/installation/versions-interoperability.html
However, it may be nice to allow users still on those versions to
continue to use changefeeds.
While we could reduce the Version constant back to its original value,
that comes at the cost of potentially making the Kafka server do more
work to handle the older protocol messages.
This change allows the user to set the version via the
kafka_sink_config option. For example, users who need to connect to
Confluent 3 would use this new option like so:
Release note (enterprise):
kafka_sink_confignow supports aversionconfiguration item to specify Kafka server versions. This islikely only necessary for old (Kafka 0.11/Confluent 3.3 or early)
Kafka servers. Further, unspecified items in
kafka_sink_confignowretain their default values.