HDDS-4980. Bump jaeger-client to 1.6.0 #2045

smengcl · 2021-03-15T18:38:22Z

https://issues.apache.org/jira/browse/HDDS-4980

Bump jaeger-client to 1.5.1 which should include libthrift 0.14.1 that addresses CVE-2020-13949.

Change-Id: I3606cb86de0811a7fb23eeb1221bcf311140fca9

Change-Id: I70d2fafb688fe3bf132cff28c57b1b93e5dd61ee

Change-Id: Iae45d9e62e8d816498ff16dbacbca453dce69fab

This reverts commit 22acdab. Change-Id: Ibd0515dcf5ca36b4b9e3ab6e2c25d38ee8564dbd

smengcl · 2021-03-15T20:54:51Z

dep tree looks good with the current patch:

$ mvn dependency:tree
...
[INFO] +- io.jaegertracing:jaeger-client:jar:1.2.0:compile
[INFO] |  +- io.jaegertracing:jaeger-thrift:jar:1.2.0:compile
[INFO] |  |  +- org.apache.thrift:libthrift:jar:0.14.1:compile
...

Change-Id: I790896b03707726cf663436d329101f096507811

smengcl · 2021-03-15T21:04:42Z

mvn build looks fine as well:

$ find hadoop-ozone/dist/target -iname "*thrift*"
hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/share/ozone/lib/libthrift-0.14.1.jar
hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/share/ozone/lib/jaeger-thrift-1.2.0.jar

adoroszlai

Thanks @smengcl for working on this. Unfortunately it doesn't seem to work with the current version of Jaeger.

cd hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/compose/ozone
export COMPOSE_FILE=docker-compose.yaml:monitoring.yaml
docker-compose up -d --scale datanode=3

Result (same for all services):

datanode_1    | java.lang.NoSuchMethodError: 'void org.apache.thrift.transport.AutoExpandingBufferWriteTransport.<init>(int, int)'
datanode_1    | 	at io.jaegertracing.thrift.internal.senders.ThriftSenderBase.<init>(ThriftSenderBase.java:67)
datanode_1    | 	at io.jaegertracing.thrift.internal.senders.ThriftSender.<init>(ThriftSender.java:41)
datanode_1    | 	at io.jaegertracing.thrift.internal.senders.UdpSender.<init>(UdpSender.java:47)
datanode_1    | 	at io.jaegertracing.thrift.internal.senders.ThriftSenderFactory.getSender(ThriftSenderFactory.java:36)
datanode_1    | 	at io.jaegertracing.internal.senders.SenderResolver.getSenderFromFactory(SenderResolver.java:110)
datanode_1    | 	at io.jaegertracing.internal.senders.SenderResolver.resolve(SenderResolver.java:88)
datanode_1    | 	at io.jaegertracing.Configuration$SenderConfiguration.getSender(Configuration.java:696)
datanode_1    | 	at io.jaegertracing.Configuration$ReporterConfiguration.getReporter(Configuration.java:593)
datanode_1    | 	at io.jaegertracing.Configuration$ReporterConfiguration.access$000(Configuration.java:553)
datanode_1    | 	at io.jaegertracing.Configuration.getTracerBuilder(Configuration.java:230)
datanode_1    | 	at org.apache.hadoop.hdds.tracing.TracingUtil.initTracing(TracingUtil.java:53)

So I guess we need to wait for new Jaeger release.

smengcl · 2021-03-16T15:13:10Z

@adoroszlai thanks for checking. i tried to bump jaeger-client from 1.2.0 to 1.5.0 but the result is the same. i guess we have to wait for a new jaeger java client release.

blocked on jaegertracing/jaeger-client-java#768

smengcl · 2021-03-23T13:16:12Z

Waiting for a new jaeger-client release.

Change-Id: I4c1aa8bc081ff2cea5c0277ee6afb84e2408fbc0

Change-Id: I5e14315c4433d37780ed29ebb6a7e5a1b58d997e

smengcl · 2021-03-31T13:09:04Z

There should be a 1.6.0 release soon:

jaegertracing/jaeger-client-java#774 (comment)

Retrigger CI when it releases.

smengcl · 2021-04-01T17:46:39Z

1.6.0 tagged but not released yet:
https://github.com/jaegertracing/jaeger-client-java/releases
https://mvnrepository.com/artifact/io.jaegertracing/jaeger-client

Should expect a release in ~1 week?

smengcl · 2021-04-07T18:22:03Z

1.6.0 released. Retriggering CI.

Change-Id: Idb8f1ada26aea215d83d58e9211f188ddc216485

adoroszlai · 2021-04-08T08:21:43Z

@smengcl Thanks for updating the patch. Tracing looks good now. I've noticed this error in S3 Gateway log:

s3g_1         | 07:46:42.267 [main] ERROR org.jboss.weld.environment.servletTomcat - WELD-ENV-001102: Unable to replace Tomcat 7 AnnotationProcessor. CDI injection will not be available in Servlets, Filters, or Listeners.
s3g_1         | java.lang.RuntimeException: WELD-ENV-001104: Cannot get StandardContext from ServletContext.
s3g_1         | 	at org.jboss.weld.environment.tomcat.WeldForwardingInstanceManager.getStandardContext(WeldForwardingInstanceManager.java:104) ~[weld-servlet-2.4.7.Final.jar:2.4.7.Final]
s3g_1         | 	at org.jboss.weld.environment.tomcat.WeldForwardingInstanceManager.replaceInstanceManager(WeldForwardingInstanceManager.java:84) ~[weld-servlet-2.4.7.Final.jar:2.4.7.Final]
s3g_1         | 	at org.jboss.weld.environment.tomcat.TomcatContainer.initialize(TomcatContainer.java:43) [weld-servlet-2.4.7.Final.jar:2.4.7.Final]
s3g_1         | 	at org.jboss.weld.environment.servlet.WeldServletLifecycle.initialize(WeldServletLifecycle.java:220) [weld-servlet-2.4.7.Final.jar:2.4.7.Final]
s3g_1         | 	at org.jboss.weld.environment.servlet.Listener.contextInitialized(Listener.java:125) [weld-servlet-2.4.7.Final.jar:2.4.7.Final]
s3g_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068) [jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
s3g_1         | 	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572) [jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
s3g_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997) [jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
s3g_1         | 	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746) [jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
...
s3g_1         | 	at org.eclipse.jetty.server.Server.doStart(Server.java:387) [jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
s3g_1         | 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
s3g_1         | 	at org.apache.hadoop.hdds.server.http.HttpServer2.start(HttpServer2.java:1187) [hadoop-hdds-server-framework-1.1.0-SNAPSHOT.jar:?]
s3g_1         | 	at org.apache.hadoop.hdds.server.http.BaseHttpServer.start(BaseHttpServer.java:295) [hadoop-hdds-server-framework-1.1.0-SNAPSHOT.jar:?]
s3g_1         | 	at org.apache.hadoop.ozone.s3.Gateway.start(Gateway.java:69) [hadoop-ozone-s3gateway-1.1.0-SNAPSHOT.jar:?]
s3g_1         | 	at org.apache.hadoop.ozone.s3.Gateway.call(Gateway.java:58) [hadoop-ozone-s3gateway-1.1.0-SNAPSHOT.jar:?]
s3g_1         | 	at org.apache.hadoop.ozone.s3.Gateway.call(Gateway.java:37) [hadoop-ozone-s3gateway-1.1.0-SNAPSHOT.jar:?]
...
s3g_1         | 	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:87) [hadoop-hdds-common-1.1.0-SNAPSHOT.jar:?]
s3g_1         | 	at org.apache.hadoop.ozone.s3.Gateway.main(Gateway.java:48) [hadoop-ozone-s3gateway-1.1.0-SNAPSHOT.jar:?]
s3g_1         | Caused by: java.lang.ClassCastException: class org.eclipse.jetty.webapp.WebAppContext$Context cannot be cast to class org.apache.catalina.core.ApplicationContextFacade (org.eclipse.jetty.webapp.WebAppContext$Context and org.apache.catalina.core.ApplicationContextFacade are in unnamed module of loader 'app')
s3g_1         | 	at org.jboss.weld.environment.tomcat.WeldForwardingInstanceManager.getStandardContext(WeldForwardingInstanceManager.java:101) ~[weld-servlet-2.4.7.Final.jar:2.4.7.Final]
s3g_1         | 	... 41 more

Change-Id: I6f3bbd1888da63b5273d4480d9c46eacd9587cf5

smengcl · 2021-04-08T23:19:04Z

@adoroszlai Interesting find. I've observed the same. Though create-bucket through S3 Gateway still works (no security):

aws s3api --endpoint http://127.0.0.1:9878 create-bucket --bucket buck1

2021-04-08 22:55:44,193 [main] INFO http.BaseHttpServer: HTTP server of s3gateway listening at http://0.0.0.0:9878
2021-04-08 22:57:42,194 [qtp1223850219-23] INFO rpc.RpcClient: Creating Bucket: s3v/buck1, with Versioning false and Storage Type set to DISK and Encryption set to false
2021-04-08 22:57:42,387 [qtp1223850219-23] INFO endpoint.BucketEndpoint: Location is /buck1

The Web UI is probably broken. When I switch back to 1.2.0, the message is gone.

smengcl · 2021-04-08T23:40:30Z

I think it's new libthrift's tomcat-embed-core dependency that messed up the class loader:

[INFO] +- io.jaegertracing:jaeger-client:jar:1.6.0:compile
[INFO] |  +- io.jaegertracing:jaeger-thrift:jar:1.6.0:compile
[INFO] |  |  +- org.apache.thrift:libthrift:jar:0.14.1:compile
[INFO] |  |  |  \- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.46:compile
[INFO] |  |  |     \- org.apache.tomcat:tomcat-annotations-api:jar:8.5.46:compile
[INFO] |  |  \- com.squareup.okhttp3:okhttp:jar:4.9.0:compile
[INFO] |  |     +- com.squareup.okio:okio:jar:2.8.0:compile
[INFO] |  |     |  \- org.jetbrains.kotlin:kotlin-stdlib-common:jar:1.4.0:compile
[INFO] |  |     \- org.jetbrains.kotlin:kotlin-stdlib:jar:1.4.10:compile
[INFO] |  |        \- org.jetbrains:annotations:jar:13.0:compile

org.apache.catalina.core.ApplicationContextFacade is included in tomcat-embed-core-8.5.46.jar

@adoroszlai we should somehow exclude that from s3 gateway / hdds common? or just shade jaeger-client entirely to prevent this

smengcl · 2021-04-12T18:28:15Z

Thanks @adoroszlai for adding the exclusion. I have verified S3 gateway startup error is gone with the exclusion of tomcat-embed-core locally in docker-compose.

HDDS-4980. Bump dependency jaeger-client version to 1.5.0

22acdab

Change-Id: I3606cb86de0811a7fb23eeb1221bcf311140fca9

smengcl changed the title ~~HDDS-4980. Bump dependency jaeger-client version to 1.5.0~~ Bump libthrift to 0.14.1 for jaeger-client / jaeger-thrift Mar 15, 2021

smengcl changed the title ~~Bump libthrift to 0.14.1 for jaeger-client / jaeger-thrift~~ HDDS-4980. Bump libthrift to 0.14.1 for jaeger-client / jaeger-thrift Mar 15, 2021

smengcl added 3 commits March 15, 2021 12:19

Upgrade libthrift to 0.14.1

dd81916

Change-Id: I70d2fafb688fe3bf132cff28c57b1b93e5dd61ee

Use dependencyManagement instead.

fbae8e3

Change-Id: Iae45d9e62e8d816498ff16dbacbca453dce69fab

Revert "HDDS-4980. Bump dependency jaeger-client version to 1.5.0"

62f4dbf

This reverts commit 22acdab. Change-Id: Ibd0515dcf5ca36b4b9e3ab6e2c25d38ee8564dbd

smengcl requested review from elek and adoroszlai March 15, 2021 20:56

Trigger CI

24942c3

Change-Id: I790896b03707726cf663436d329101f096507811

adoroszlai requested changes Mar 16, 2021

View reviewed changes

smengcl changed the title ~~HDDS-4980. Bump libthrift to 0.14.1 for jaeger-client / jaeger-thrift~~ HDDS-4980. [Blocked] Bump jaeger-client to 1.5.1 Mar 23, 2021

adoroszlai marked this pull request as draft March 23, 2021 14:52

mukul1987 force-pushed the master branch 2 times, most recently from 79a9d39 to 520ba00 Compare March 25, 2021 16:05

smengcl added 2 commits March 25, 2021 13:09

1.5.1-SNAPSHOT for testing.

c312a4c

Change-Id: I4c1aa8bc081ff2cea5c0277ee6afb84e2408fbc0

jaeger-client 1.6.0

b7b4e1d

Change-Id: I5e14315c4433d37780ed29ebb6a7e5a1b58d997e

smengcl changed the title ~~HDDS-4980. [Blocked] Bump jaeger-client to 1.5.1~~ HDDS-4980. [Blocked] Bump jaeger-client to 1.6.0 Mar 31, 2021

smengcl changed the title ~~HDDS-4980. [Blocked] Bump jaeger-client to 1.6.0~~ HDDS-4980. Bump jaeger-client to 1.6.0 Mar 31, 2021

smengcl marked this pull request as ready for review April 7, 2021 18:22

Retrigger CI

80e9d48

Change-Id: Idb8f1ada26aea215d83d58e9211f188ddc216485

CI

56ee77f

Change-Id: I6f3bbd1888da63b5273d4480d9c46eacd9587cf5

adoroszlai added 2 commits April 9, 2021 19:13

Exclude tomcat-embed-core

e034005

Merge remote-tracking branch 'origin/master' into HDDS-4980

8590f7d

adoroszlai approved these changes Apr 9, 2021

View reviewed changes

smengcl merged commit fd308b7 into apache:master Apr 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-4980. Bump jaeger-client to 1.6.0 #2045

HDDS-4980. Bump jaeger-client to 1.6.0 #2045

smengcl commented Mar 15, 2021 •

edited

Loading

smengcl commented Mar 15, 2021

smengcl commented Mar 15, 2021

adoroszlai left a comment

smengcl commented Mar 16, 2021

smengcl commented Mar 23, 2021

smengcl commented Mar 31, 2021

smengcl commented Apr 1, 2021

smengcl commented Apr 7, 2021

adoroszlai commented Apr 8, 2021

smengcl commented Apr 8, 2021 •

edited

Loading

smengcl commented Apr 8, 2021 •

edited

Loading

smengcl commented Apr 12, 2021

HDDS-4980. Bump jaeger-client to 1.6.0 #2045

HDDS-4980. Bump jaeger-client to 1.6.0 #2045

Conversation

smengcl commented Mar 15, 2021 • edited Loading

smengcl commented Mar 15, 2021

smengcl commented Mar 15, 2021

adoroszlai left a comment

Choose a reason for hiding this comment

smengcl commented Mar 16, 2021

smengcl commented Mar 23, 2021

smengcl commented Mar 31, 2021

smengcl commented Apr 1, 2021

smengcl commented Apr 7, 2021

adoroszlai commented Apr 8, 2021

smengcl commented Apr 8, 2021 • edited Loading

smengcl commented Apr 8, 2021 • edited Loading

smengcl commented Apr 12, 2021

smengcl commented Mar 15, 2021 •

edited

Loading

smengcl commented Apr 8, 2021 •

edited

Loading

smengcl commented Apr 8, 2021 •

edited

Loading