Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opentelemetry VertxTracingAdapter creates duplicate and high cardinality spans for requests #18248

Closed
hacst opened this issue Jun 29, 2021 · 4 comments
Labels

Comments

@hacst
Copy link
Contributor

hacst commented Jun 29, 2021

Describe the bug

Kind of a follow-up for #16952 . When using the opentelemetry extension and observing HTTP spans created for a rest-client request Quarkus creates two client spans. One based on rest-client knowledge in ClientTracingFilter with a proper templated name and another span basically duplicating the information at the vertx level in VertxTracingAdapter. The duplicate span also is not aware of the template so it uses the full name instead (#16952 explains why that is problematic).

Interestingly enough it looks like the Kafka extension encountered a similar problem and special handling was introduced:

if (policy.equals(TracingPolicy.IGNORE)) {
/*
* SmallRye Reactive Messaging with Kafka is responsible for creating spans for outgoing messages.
* In this SPI call there is no way to know whether it has happened.
* The current approach to prevent duplicate spans is for the Kafka client in SmallRye Reactive Messaging
* to set TracingPolicy.IGNORE
* Disabling tracing in Quarkus will not be done with TracingPolicy.IGNORE,
* allowing it to be used for this purpose.
*/
return null;
}

I cannot find where the Kafka extension actually uses this though.

Expected behavior

Ideally only one client span is created for each outgoing request. The span has the template known from the rest-client as its name.

Actual behavior

Two spans are created. One of which has a high cardinality name equal to the requested URL.

To Reproduce

A test sending a rest client request will reproduce the behavior.

Quarkus version or git rev

2.0.0.Final

@hacst hacst added the kind/bug Something isn't working label Jun 29, 2021
@quarkus-bot
Copy link

quarkus-bot bot commented Jun 29, 2021

/cc @Ladicek, @kenfinnigan

@hacst
Copy link
Contributor Author

hacst commented Jun 29, 2021

Another thing I noticed is that the http.url attribute ClientTracingFilter sets contains the userinfo portion of the URL to which https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md#common-attributes states "http.url MUST NOT contain credentials passed via URL in form of https://username:[email protected]/.". Not 100% they also mean clients there but I think it is both. I can open another issue for this if required.

PS: Very sorry for reporting this so late. I'm very aware that I should've tested beforehand but I only now found some time.

@kenfinnigan
Copy link
Member

@hacst please provide a reproducer as I've used tracing with REST Client and not seen duplicate spans.

@hacst
Copy link
Contributor Author

hacst commented Jun 29, 2021

Turns out the problem was on my end. Now everything works as intended. (the url.http containing credentials is still true though)

@hacst hacst closed this as completed Jun 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants