-
Notifications
You must be signed in to change notification settings - Fork 25.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update OAuth2 OIDC SDK #108799
Update OAuth2 OIDC SDK #108799
Conversation
This reverts commit 893495d.
@rjernst @jakelandis This is the PR in question with the security manager issues, run the failing tests with It's possible that I've misunderstood something - I don't feel like I have a great handle on this stuff, so please if something looks off do call it out. |
Looking at the stack trace:
Based on what you described, I suspect the problematic frames are those from the server jar, like this:
Perhapse the doPrivileged needs to be moved into this JwtRealm lambda?
|
# Conflicts: # x-pack/plugin/core/build.gradle
|
...security/lib/nimbus-jose-jwt-fixed/src/main/java/com/nimbusds/jose/util/JSONStringUtils.java
Outdated
Show resolved
Hide resolved
@@ -266,7 +266,7 @@ public void testJwtSignVerifyPassedForAllSupportedAlgorithms() { | |||
try { | |||
helpTestSignatureAlgorithm(signatureAlgorithm, false); | |||
} catch (Exception e) { | |||
fail("signature validation with algorithm [" + signatureAlgorithm + "] should have succeeded"); | |||
throw new RuntimeException("signature validation with algorithm [" + signatureAlgorithm + "] should have succeeded", e); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: this could still be AssertionError
* 3) This comment and the license comment | ||
*/ | ||
@SuppressWarnings({ "unchecked", "rawtypes" }) | ||
public class JSONObjectUtils { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand the desire to avoid lots of doPrivileged calls. However, this seems quite a bit more fragile than other times we have replaced classes from dependencies. For example, with the hdfs client we replaced the shutdown manager with noop methods. In contrast, here many method implementations are copied, and if in future updates they are modified, we might still have the old implementation. It seems like a bigger risk, I lean towards having the doPrivileged calls (though admittedly I did not look at the previous PR iteration, so I don't know how pervasive the doPrivs were).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's less that the doPriv calls were pervasive and more that a lot of things end up calling into one of these methods, including constructors and toString
methods, to the point where it's not just "pervasive calls", it's "it's legit really hard to add any new code without breaking something".
To take an example @n1v0lg provided (thanks!), consider these two lines:
logger.info("Claim set small whoops: {}", jwtAuthenticationToken.getJWTClaimsSet());
logger.info("Claim set big whoops: " + jwtAuthenticationToken.getJWTClaimsSet());
Small whoops ends up logging a message like:
Claim set small whoops: [!!!com.nimbusds.jwt.JWTClaimsSet@15ee0d6d=>java.security.AccessControlException:access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")!!!]
Big whoops throws an exception. It's very, very, very easy to accidentally hit this, and not even necessarily cause a test failure about it. While we can work around that with forbiddenAPIs (which is what I started doing and can revert back to if we would really, really rather do it that way), it's a huge pain in the butt now and any time we have to do anything with the JWT or OIDC realms going forward. On top of that, the forbidden APIs list (note: that's just nimbus-jose-jwt, one of two libraries, I still have to do a list for oauth2-oidc) is made by me going through the libraries in question with IntelliJ, tracing all the various ways that JSONObjectUtils
can be called into. This would need to be updated manually every time we update these dependencies, and less obviously so - at least this is likely to trigger a compilation or test failure if there are significant changes, whereas a forbiddenAPIs-based solution not being properly updated when the dependency is would result in silently permitting us to ship one or both of the "whoops" cases above.
All the options suck. I'm open to being talked into the idea that the forbiddenAPIs-based solution sucks less than vendoring, but it's not good.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I generally like this approach better. However, rather than removing the original class then replacing functionality with wrap(copy/paste), can we instead move the original class to different package then replace the functionality with wrap(delegate(method)) ?
So instead of duplicating the source code, we keep their original source but in a different package name and just wrap the calls to the original code inside the doPrivilged ? It should surface any Java API breaking changes to these methods as compile time issues the next version bump.
This won't help with the 2 private static method/members, but those seem unlikely to change and we if we really want to (i don't think it is necessary) we could use the elevated privs to delegate to them too via reflections. It might worth a unit test or two to use reflections and mocks to help ensure the same behavior between the original and what is copied, or maybe just a comment in the build.gradle to double check those on a version bump.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ooh, I like that idea, I think it addresses the worst of the concerns Ryan raised. I'll get that spun together, thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jakelandis I've put together an approach with JSONObjectUtils
(and JSONStringUtils
) copied verbatim (sans formatting) into InnerJSONObjectUtils
(& InnerJSONStringUtils
) and produced wrappers under the original names. It's not quite as clean as it maybe could be since everything is static, but I think it's still an improvement.
* @version 2022-08-19 | ||
*/ | ||
@SuppressWarnings({ "unchecked", "rawtypes" }) | ||
public class InnerJSONObjectUtils { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than copy the source, we should be able to rename the existing classfile when pulling in the dependency?
https://gradleup.com/shadow/configuration/relocation/#filtering-relocation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't shadowJar rewrite the rest of the JAR to "correctly" reference the moved class when doing that? If I've misunderstood on that point please do correct me, I think that would be better it's just not how I understood that part of shadowJar to work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are right, it does. However, I think we can still workaround that. We essentially want 3 parts to the final jar:
- The original jar minus these two classes
- These two classes renamed
- The two replacement classes you are adding in
The first two bullets can be separate shadowjar tasks, and ultimately everthing from these 3 parts will go into jar.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aha, so essentially several intermediate gradle tasks to build a few temporary JARs with the right properies, then merge them into a single final target? Yeah, I think I see how that can work, let me give it a shot.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rjernst This is done, I agree this is an improvement even though shadowJar gave me a bit of a headache about it. Thanks!
// Attempting to exclude all of the classes we *don't* move here ought to be possible per the | ||
// shadowJar docs, but actually attempting to do so results in an empty JAR. So we'll do that as a | ||
// separate step (see njj-moved-utils-only) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seriously, I spent some quality time with the shadowJar source trying to figure out why all the include-based strategies to define which files to include resulted in an empty JAR, and exclude-based ones would require listing out a bunch of files individually since there's a bunch of other stuff in the com.nimbusds.jose.util
package. I still don't understand why; I think there must be some bug with the relocate implementation and include directives. Breaking it out into two steps like this solves the problem relatively cleanly, but I welcome anyone who can figure out the correct incantation to do the right thing in one step - I think it should possible, absent bugs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To clarify, putting exclude 'com/nimbusds/'
in this task also results in an empty JAR, even though it's exclude-based, regardless of where it's put in the process.
@@ -11,7 +11,6 @@ | |||
import com.nimbusds.jose.jwk.JWK; | |||
import com.nimbusds.jose.jwk.JWKSet; | |||
import com.nimbusds.jose.util.Base64URL; | |||
import com.nimbusds.jose.util.JSONObjectUtils; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IntelliJ wasn't happy about this, I'm not quite sure why - Gradle didn't mind. Easy enough to replace, and none of the other classes seemed to have an issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good just a couple minor comments.
} catch (PrivilegedActionException e) { | ||
if (e.getException() instanceof ParseException pe) { | ||
throw pe; | ||
} else if (e.getException() instanceof RuntimeException re) { | ||
throw re; | ||
} | ||
throw new RuntimeException(e); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: javadoc:
This exception is thrown by doPrivileged(PrivilegedExceptionAction) and doPrivileged(PrivilegedExceptionAction, AccessControlContext context) to indicate that the action being performed threw a checked exception. The exception thrown by the action can be obtained by calling the getException method. In effect, an PrivilegedActionException is a "wrapper" for an exception thrown by a privileged action.
A literal reading of that means we won't ever catch a RuntimeException here ("threw a checked exception"), so no need for the else
or throw new RuntimeException
, only unwrapping the PrivilegedActionException to rethrow it (and no need to check the type)
so I think we only need:
} catch (PrivilegedActionException e) {
throw e.getException();
}
same for the rest.
// 2) Create a JAR from the result of step 1 which contains *only* the relevant class files. | ||
// 3) Use the result of step 2 here, combined with the original library, so that we can use our | ||
// replacement classes which wrap the original class files. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I am +1 to what ever works in the build and I don't think spending more time fighting the build for a more elegant is implementation needed. However, can you clean up the comments and naming a bit ?
- can you add the comments to the empty build.gradle and then each of the 3 sub projects add a small //See parent build.gradle for addition information about this unique build.
- can you name the other gradle projects "nimbus-jose-jwt-modified-build-part1" and "nimbus-jose-jwt-modified-build-part2" or the like..the point being the same name as this jar where the order and purpose is easy to figure out by the name (and ideally use those names in the doc)
@@ -0,0 +1,43 @@ | |||
/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: can we call this modified
instead of fixed
? Fixed implies we fixed a bug, but really we just modified the original to support the security manager.
Thanks for the review @jakelandis, I've adjusted the exception handling and directory names. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
thanks for the iterations!
A quick summary of why this version bump was so involved:
|
@mark-vieira Apologies, feel free to disregard that review, misclicked. (Of course, if you have feedback, it's welcome as always.) |
* upstream/main: (91 commits) Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasticsearch.xpack.test.rest.XPackRestIT elastic#111944 Add audit_unenrolled_* attributes to fleet-agents template (elastic#111909) Fix windows memory locking (elastic#111866) Update OAuth2 OIDC SDK (elastic#108799) Adds a warning about manually mounting snapshots managed by ILM (elastic#111883) Update geoip fixture files and utility methods (elastic#111913) Updated Function Score Query Test with Explain Fixes for 8.15.1 (elastic#111929) Mute org.elasticsearch.xpack.sql.qa.security.JdbcCsvSpecIT org.elasticsearch.xpack.sql.qa.security.JdbcCsvSpecIT elastic#111923 [ESQL] date nanos binary comparisons (elastic#111908) [DOCS] Documents output_field behavior after multiple inference runs (elastic#111875) Add additional BlobCacheMetrics, expose BlobCacheMetrics via SharedBlobCacheService (elastic#111730) Mute org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT elastic#111923 Mute org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT test {agg-ordering.testHistogramDateTimeWithCountAndOrder_2} elastic#111919 Mute org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT test {date.testDateParseHaving} elastic#111921 Mute org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT test {agg-ordering.testHistogramDateTimeWithCountAndOrder_1} elastic#111918 Mute org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT test {datetime.testDateTimeParseHaving} elastic#111922 Mute org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT elastic#111923 Mute org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT test {agg-ordering.testHistogramDateTimeWithCountAndOrder_1} elastic#111918 Mute org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT test {datetime.testDateTimeParseHaving} elastic#111922 Mute org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT test {date.testDateParseHaving} elastic#111921 ... # Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java
This commit updates the Nimbus OAuth2 OIDC SDK and the associated Nimbus JOSE+JWT, however a few odd choices had to be made in the process. First, we update to versions which are old at time of merge. This is because versions of Nimbus JOSE+JWT 9.38 and after through time of writing [contain a bug](https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/550/java-module-doesnt-work-properly-with) in which the shaded gson class files included in the library are not properly loaded by our module loading code (and possibly in general? the root cause of the bug is unclear at time of writing but it does not appear to be present in all uses of this library). This requires us to use an older version of Nimbus OAuth2 OIDC SDK as well. Second, the aforementioned shaded gson uses reflection internally, and is used in unpredictable places in these libraries (e.g. constructors). This is extremely unfriendly to our usage of the security manager. In order to make the scope of permission grants as narrow as possible, we shadow nimbus-jose-jwt in order to insert `AccessController.doPrivileged` calls at the appropriate points, given the usage of gson is relatively contained. This approach was chosen over other approaches given 1) the relative simplicity given the implementation of the library, and 2) the complexity involved in safely using the library any other way - as one example, gson is used frequently in `toString()` methods, which are frequently called implicitly, especially in combination with logging which may mask security manager exceptions from being surfaced in tests. All of the code we intercept should be re-evaluated when this library is next upgraded. Co-authored-by: Jake Landis <[email protected]>
This commit updates the Nimbus OAuth2 OIDC SDK and the associated Nimbus JOSE+JWT, however a few odd choices had to be made in the process. First, we update to versions which are old at time of merge. This is because versions of Nimbus JOSE+JWT 9.38 and after through time of writing [contain a bug](https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/550/java-module-doesnt-work-properly-with) in which the shaded gson class files included in the library are not properly loaded by our module loading code (and possibly in general? the root cause of the bug is unclear at time of writing but it does not appear to be present in all uses of this library). This requires us to use an older version of Nimbus OAuth2 OIDC SDK as well. Second, the aforementioned shaded gson uses reflection internally, and is used in unpredictable places in these libraries (e.g. constructors). This is extremely unfriendly to our usage of the security manager. In order to make the scope of permission grants as narrow as possible, we shadow nimbus-jose-jwt in order to insert `AccessController.doPrivileged` calls at the appropriate points, given the usage of gson is relatively contained. This approach was chosen over other approaches given 1) the relative simplicity given the implementation of the library, and 2) the complexity involved in safely using the library any other way - as one example, gson is used frequently in `toString()` methods, which are frequently called implicitly, especially in combination with logging which may mask security manager exceptions from being surfaced in tests. All of the code we intercept should be re-evaluated when this library is next upgraded. Co-authored-by: Jake Landis <[email protected]>
This commit updates the Nimbus OAuth2 OIDC SDK and the associated Nimbus JOSE+JWT, however a few odd choices had to be made in the process. First, we update to versions which are old at time of merge. This is because versions of Nimbus JOSE+JWT 9.38 and after through time of writing [contain a bug](https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/550/java-module-doesnt-work-properly-with) in which the shaded gson class files included in the library are not properly loaded by our module loading code (and possibly in general? the root cause of the bug is unclear at time of writing but it does not appear to be present in all uses of this library). This requires us to use an older version of Nimbus OAuth2 OIDC SDK as well. Second, the aforementioned shaded gson uses reflection internally, and is used in unpredictable places in these libraries (e.g. constructors). This is extremely unfriendly to our usage of the security manager. In order to make the scope of permission grants as narrow as possible, we shadow nimbus-jose-jwt in order to insert `AccessController.doPrivileged` calls at the appropriate points, given the usage of gson is relatively contained. This approach was chosen over other approaches given 1) the relative simplicity given the implementation of the library, and 2) the complexity involved in safely using the library any other way - as one example, gson is used frequently in `toString()` methods, which are frequently called implicitly, especially in combination with logging which may mask security manager exceptions from being surfaced in tests. All of the code we intercept should be re-evaluated when this library is next upgraded. Co-authored-by: Jake Landis <[email protected]>
This commit updates the Nimbus OAuth2 OIDC SDK and the associated Nimbus JOSE+JWT, however a few odd choices had to be made in the process.
First, we update to versions which are old at time of merge. This is because versions of Nimbus JOSE+JWT 9.38 and after through time of writing contain a bug in which the shaded gson class files included in the library are not properly loaded by our module loading code (and possibly in general? the root cause of the bug is unclear at time of writing but it does not appear to be present in all uses of this library). This requires us to use an older version of Nimbus OAuth2 OIDC SDK as well.
Second, the aforementioned shaded gson uses reflection internally, and is used in unpredictable places in these libraries (e.g. constructors). This is extremely unfriendly to our usage of the security manager. In order to make the scope of permission grants as narrow as possible, we shadow nimbus-jose-jwt in order to insert
AccessController.doPrivileged
calls at the appropriate points, given the usage of gson is relatively contained.This approach was chosen over other approaches given 1) the relative simplicity given the implementation of the library, and 2) the complexity involved in safely using the library any other way - as one example, gson is used frequently in
toString()
methods, which are frequently called implicitly, especially in combination with logging which may mask security manager exceptions from being surfaced in tests. All of the code we intercept should be re-evaluated when this library is next upgraded.