Issues establishing TLS connection to secure Kafka

[karatekidandy]

Requirement - what kind of business use case are you trying to solve?

We have a secure Kafka cluster running 5.2.2 (we have also tested on 5.3.2)
and are using the tls settings in Jaeger to connect to the cluster. However, it doesn't appear to be working

Problem - what in Jaeger blocks you from solving the requirement?

If we set the kafka client authentication parameter to 'Required' from 'Requested', we get an SSL handshake error in Kafka and a 'client has run out of available brokers' error from the Jaeger log. We believe the certificates are configured correctly.

Any help/guidance would be greatly appreciated.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Any open questions to address

[pavolloffay]

@karatekidandy hi, what version of Jaeger are you using? Could you please share Jaeger collector/ingester configuration?

cc @rubenvp8510

[karatekidandy]

Version 1.16

Collector:
jaeger-collector --collector.zipkin.http-port=9411 --kafka.producer.authentication=tls --kafka.producer.tls.ca=/opt/jaeger/project.crt --kafka.producer.tls.cert=/opt/jaeger/jaeger.crt --kafka.producer.tls.key=/opt/jaeger/jaeger.key --kafka.producer.brokers='${kafka_brokers}' --kafka.producer.topic=jaeger --kafka.producer.encoding=json

Ingester:
jaeger-ingester --ingester.deadlockInterval=0 --ingester.parallelism=2000 --kafka.consumer.authentication=tls --kafka.consumer.tls.ca=/opt/jaeger/project.crt --kafka.consumer.tls.cert=/opt/jaeger/jaeger.crt --kafka.consumer.tls.key=/opt/jaeger/jaeger.key --kafka.consumer.brokers='${kafka_brokers}' --kafka.consumer.topic=jaeger --kafka.consumer.group-id=jaeger-ingester --kafka.consumer.encoding=json --es.server-urls=http://${elasticsearch_url}:80 --es.num-shards=6 --es.num-replicas=1 --es.max-span-age=168h0m0s --es.tags-as-fields.all=true

[karatekidandy]

Hi there, did the config help?

[rubenvp8510]

Hi @karatekidandy yes, I was able to reproduce the issue, I'm investigating the root cause, will go back with my findings.

Thanks

[karatekidandy]

@rubenvp8510 great thank you!

[karatekidandy]

@rubenvp8510 Hey ruben, any update on the root cause/potential fix? Thanks!

[rubenvp8510]

Hi @karatekidandy sorry for late reply

After look this more closely, I wasn't able to reproduce the issue on 1.16,

Those are the configurations, commands for generate certificates and for launch ingester/collector that I used on my tests:

https://gist.github.com/rubenvp8510/34f7316e1dca6d4e73d654407efb1760

Hope this will help you.

The reason I thought I was able to replicate was because I tested using master (also tested on 1.17). If you are using the latest version you need to add a flag to enable TLS.

• for the collector: --kafka.producer.tls.enabled=true
• for the ingester --kafka.consumer.tls.enabled=true )

Something like this:
https://gist.github.com/rubenvp8510/30f3a7beb5d86dd3f5d06ced0d1753d9

This was tested on kafka 2.12 which I think is what confluence platform 5.2.2 uses.

[karatekidandy]

Thanks @rubenvp8510 will give this a go :) i appreciate your time working on this.

[pavolloffay]

@karatekidandy were you able to resolve this issue? Can we close this ticket?