JWT Token builder bottleneck for internal communication #10889
Comments
|
Thank you for reporting this and for a great analysis. We will take a look into it. |
|
Thanks for the detailed report. It looks like a regression introduced by |
|
Actually it's only part of the problems caused by |
|
We found few additional problematic places which affects cluster performance Problematic parts in Jwts.java public static JwtParserBuilder parserBuilder() {
return (JwtParserBuilder)Classes.newInstance("io.jsonwebtoken.impl.DefaultJwtParserBuilder");
}
public static JwtBuilder builder() {
return (JwtBuilder)Classes.newInstance("io.jsonwebtoken.impl.DefaultJwtBuilder");
}And DefaultCompressionCodecResolver.java also scanning Jar files too public DefaultCompressionCodecResolver() {
Map<String, CompressionCodec> codecMap = new HashMap();
Iterator i$ = Services.loadAll(CompressionCodec.class).iterator(); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
After upgrade from 346 to 367 we started to experience significant performance issues. Number of currently running queries dropped from 120 to 30-70 (green line). Number of blocked queries increased from 0-1 to 10-50 (purple line).
We analyzed Java Flight Recorder and Thread dumps from coordinator. JFR shows that
DefaultJwtBuilder.compact()takes about 10% of CPU.There is a method
LegacyService.loadFirst()which is called during token generation and which is the most expensive:According to source code of the library:
https://github.com/jwtk/jjwt/blob/82189f8418d9bacad8b7df42a08ec203dc2df519/impl/src/main/java/io/jsonwebtoken/impl/DefaultJwtBuilder.java#L296-L301
This method is called to lookup for
Serializerif there is no default one. And it seems to do it by scanning jar files.Thread dump analysis showed that it's a bottleneck because 341 threads are blocked on
java.io.RandomAccessFile:Here is a stack trace of blocking thread:
The situation on workers are similar, 73 threads are blocked on
java.io.RandomAccessFileStackTrace of blocking thread:
To prove the issue we have changed the code of InternalAuthenticationManager
trino/core/trino-main/src/main/java/io/trino/server/InternalAuthenticationManager.java
Lines 116 to 133 in 27728e6
we added default serialzier
and deserializer
It fixed our problem, running queries is ~120 and stable and there is no blocked.
Jwts.builder()is also used in OAuth2Service, FormWebUiAuthenticationFilter, JsonWebTokenHandler and may also cause some issues but we haven't investigated it.The text was updated successfully, but these errors were encountered: