-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kafka Streams SASL and SSL config #7417
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@alesj, a few comments inline. Apart from those: the docs (props table in the guide) should be updated, too. Plus, ideally the existing integration test would at least cover one simple SSL set-up, so to make sure things work in general and future regressions are avoided.
* Login thread will sleep until the specified window factor of time from last refresh | ||
*/ | ||
@ConfigItem(name = "kerberos.ticket.renew.window.factor") | ||
public Optional<String> kerberosTicketRenewWindowFactor; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this a string, shouldn't it be rather a double (some for some more props below)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I didn't bother, since KS's config takes care of the conversions anyway.
If you think we should, then this can easily be changed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's ask @gsmet. My understanding is that Quarkus wants to expose properly typed options, so to offer e.g. editor support (validation) down the road.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the Quarkus config should be properly typed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it should be typed.
* The amount of buffer time before credential expiration to maintain when refreshing a credential | ||
*/ | ||
@ConfigItem(name = "login.refresh.buffer.seconds") | ||
public Optional<String> loginRefreshBufferSeconds; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be an int. But also worth double checking with Quarkus guidelines: should it be rather Duration
, and be given as such (which would impact the option name, though, which might not be desirable as it's coming from Kafka upstream IIUC).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah all these properties should be Duration
and we should drop the Seconds
part. They will be documented so it's OK.
// reflection hack ... no other way to get raw props ... | ||
Field configField = KafkaStreams.class.getDeclaredField("config"); | ||
configField.setAccessible(true); | ||
StreamsConfig config = (StreamsConfig) configField.get(streams); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about adding a package visible accessor?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you mean?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, seems you get a field from the upstream KafkaStreams
object. I thought about having a getter in our own object (topology manager), where we could expose a package-visible getter for testing purposes.
I thought that's the beauty of having this config stuff, so it's auto-generated? |
Yeah, working on it. |
Ah, I didn't know that :) If so, that's much cooler of course. |
Right, @gsmet ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added some comments.
As for the documentation, it will be automatically generated so no need to do anything except having proper Javadoc.
/** | ||
* The schema registry url. | ||
*/ | ||
@ConfigItem(name = "schema.registry.url") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In Quarkus config, we use a dot to separate sections and a dash to separate words. It's a general comment about the config key of this PR.
* Login thread will sleep until the specified window factor of time from last refresh | ||
*/ | ||
@ConfigItem(name = "kerberos.ticket.renew.window.factor") | ||
public Optional<String> kerberosTicketRenewWindowFactor; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it should be typed.
* The amount of buffer time before credential expiration to maintain when refreshing a credential | ||
*/ | ||
@ConfigItem(name = "login.refresh.buffer.seconds") | ||
public Optional<String> loginRefreshBufferSeconds; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah all these properties should be Duration
and we should drop the Seconds
part. They will be documented so it's OK.
I just pushed some more work, but it's not done yet ... so no new review yet needed. |
34c8025
to
d4b1fbc
Compare
@gunnarmorling @gsmet I changed to typed props, used dash, added ssl test ... please re-review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@alesj, sorry for the delay. LGTM overall. One minor suggestion inline. And one question on the test: wouldn't the test Kafka be needed to configured accordingly, so to accept SSL connections? Maybe I'm missing something?
@@ -37,10 +37,42 @@ | |||
@ConfigItem | |||
public List<String> topics; | |||
|
|||
/** | |||
* The schema registry key. | |||
* e.g. to diff between different registry impls / instances |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps clarify the value to be used for Apicurio?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, will do.
@gunnarmorling > wouldn't the test Kafka be needed to configured accordingly, so to accept SSL connections? Maybe I'm missing something? So you're saying I should fix the Kafka tests as well, not just Kafka Streams tests? |
No, I meant Kafka Streams tests. But they launch Apache Kafka, whose launch configuration I'd expect to be adjusted so to make use of SSL. Maybe I'm missing something? |
This should be done, otherwise it wouldn't work ... |
@gunnarmorling see changes to KafkaTestResource, its properties |
Got it, yeah, that's what I was looking for. Not sure how I missed it before :) Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM; thanks, @alesj!
Hey @gsmet, it looks good to me; perhaps you might double-check on Quarkus stylistics and merge? Thx! |
Hi, I've just built a test that is representative for our environment which uses the confluent schema registry and sasl plaintext for authentication. I was able to configure a working example using the sasl and schema registry runtime properties. I'm not yet able to check if it also works as a native image, because I have trouble building a native image that works with avro and the confluent schema registry. |
Thanks for reporting back; re the issues with native, you wouldn't have to add the registry itself to the native image, this could remain separately. In terms of the actual converters, it'd be interesting to see what problems you encounter. In particular, I'd recommend to check out the serializers from the Apicurio project (which is another schema registry); they are compatible with other registries like Confluent's, too, and I could see us adding support for them by means of flipping the right switches in this extension or maybe the Quarkus Kafka core extension. CC @alesj. |
Apicurio registry: |
For the native image i'll open a different issue at some later time. I don't see us switch away from confluent's SR and (de)serializer but I'm very interested in making it work with native. I'll let you know what issues I run into and perhaps a minimal example to reproduce at a later time (several weeks from now). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @alesj ! Sorry for the delay, things have been crazy until now and I'm now looking at the backlog.
I rebased your PR and also added 2 commits.
I added one question inline about some weird filtering you did?
I have one big issue though, I'm not sure we pass the config to Kafka Streams appropriately:
2020-04-28 15:38:12,392 WARN [io.qua.config] (main) Unrecognized configuration key "quarkus.kafka-streams.security.protocol" was provided; it will be ignored
2020-04-28 15:38:13,066 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'sasl.kerberos-ticket-renew-jitter' was supplied but isn't a known config.
2020-04-28 15:38:13,066 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'sasl.login.refresh.buffer.seconds' was supplied but isn't a known config.
2020-04-28 15:38:13,066 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'ssl.endpoint-identification-algorithm' was supplied but isn't a known config.
2020-04-28 15:38:13,066 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'topics' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'bootstrap-servers' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'sasl.login-refresh-buffer' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'schema-registry-key' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'ssl.truststore.type' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'sasl.kerberos.ticket.renew.jitter' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'apicurio.registry.url' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'ssl.truststore.location' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'ssl.truststore.password' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'schema-registry-url' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'application-id' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'some-property' was supplied but isn't a known config.
2020-04-28 15:38:13,067 WARN [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'ssl.endpoint.identification.algorithm' was supplied but isn't a known config.
I thought it might be due to one of my commits but I have the same issue when going back to your commit.
The first Quarkus warning is probably just something that needs to be removed or adjusted in the application.properties but all the Kafka errors look weird to me.
Could you check what's going on? I'll be there to get this thing merged into 1.5.
Thanks!
|
||
setProperty(ssl.keymanagerAlgorithm, streamsProperties, SslConfigs.SSL_KEYMANAGER_ALGORITHM_CONFIG); | ||
setProperty(ssl.trustmanagerAlgorithm, streamsProperties, SslConfigs.SSL_TRUSTMANAGER_ALGORITHM_CONFIG); | ||
setProperty(ssl.endpointIdentificationAlgorithm.map(s -> "\"\"".equals(s) ? "" : s), streamsProperties, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand why we need that here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no other (or better / smarter) way to pass-in an empty string.
(that's what I've been told by @dmlloyd )
And in this case you need to have an empty string, to disable this config/setting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK. Can't say I'm excited about it but can't think of a better solution. Or maybe we could use -1
to disable? It's a bit of dark magic but could be documented.
Just a proposal, WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a string ... so setting a number ... hmmm, dunno :-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no other (or better / smarter) way to pass-in an empty string.
(that's what I've been told by @dmlloyd )
No it's not; I definitely never told you that. If you want a string which can be empty, use Optional<String>
with orElse("")
. If you want an integer which can be empty, use OptionalInt
(int
has no native empty representation though, so orElse
won't help in this case).
And in this case you need to have an empty string, to disable this config/setting.
Then it should be Optional
.
This somehow explains this: What you see there is what we also pass into AdminClient creation -- same props from application.properties. |
@alesj so in a normal setup, you're saying you won't see all those warnings? OK, that goes beyond my knowledge. @gunnarmorling @cescoffier WDYT of ^ |
But the optional part looks good now. |
d3d550b
to
2040dc0
Compare
Unfortunately, this won't make it. There's something not working on CI, not sure where it comes from. We have this message going on and on:
@rquinio any chance you could have a look at that one and drive that puppy home? |
I've reproduced locally the "SSL handshake failed" issue on graalvm-ce-java11-20.0.0 (but it works fine on openjdk-11.0.5), it seems systematic. Was the native test passing before the big rebase ? Btw I think we should put a JUnit timeout on the integration test, since liveness/readiness is not checked externally to shutdown KafkaStreams when something goes wrong, it'll fail faster than waiting for the 65min timeout of the CI. |
We'd really need the ability to configure kafka-streams.* properties at runtime in Quarkus 1.6, because some values can vary between platforms and cannot be known at build time. Unfortunately I don't know how to investigate the "SSL handshake failed" failure that only occurs in native mode... Other extensions like raw Kafka also have SSL native tests which work fine... An alternative could be to split the PR (ability to configure at runtime / new SSL config & tests). |
Hello About the SSL handshake failed, I have some findings : with the command :
here's what I am seeing :
It seems the SSL layer complains that a non SSL connection is being made upon the the server instance. Actually looking at the logs of the native compilation we can see the following input :
Which make me think that the configuration property from the application.properties has been merely ignored, hence the error at runtime. I will try to continue investigation when I have time Regards |
Hello Finally I think my feeling was good, there's was an issue with the kafka streams settings "security.protocol". So the native integration test was failing : the kafka streams application wanted to connect using unsecured channel, hence the SSL handshake failed on the kafka broker. I made a commit on my fork on top of @alesj contribution => here The remaining CI issue (native Tests - Data 5) is present in both CIs completion, this pull request and on my fork, so I did not bother with it, it's not linked to the content of this PR. @gsmet : what do you think? can you cherry-pick my commit in order to solve the issue and make this PR move forward? Regards |
In any case,we'll need this PR rebased onto master to solve the merge conflict |
@geoand : if I may, can you force a rebase on the PR branch ? The merge conflict is easy to solve. |
PR updated |
Thanks, @geoand. |
- Add security.protocol as quarkus runtime options and allow replaying the option as runtime init - Enable enableAllSecurityServices when building the native image for the integration test
@vietk I cherry-picked your commit and pushed it into this PR. However I don't have much Kafka experience, so I won't be reviewing the PR. I'll leave that to @gunnarmorling :) |
@gunnarmorling all checks passed ! Since this PR has already been reviewed and I carried on small changes to make it pass, can you please give a final review ? |
Let's get this in and make progress from there. |
Thanks everyone and sorry it took so much time to get in. I think a few things still need to be refined but we can do it iteratively. |
Fix #4961