Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Searchable snapshots tests fail to start secureHdfsFixture on Java 17 #78423

Closed
mark-vieira opened this issue Sep 28, 2021 · 13 comments
Closed

[CI] Searchable snapshots tests fail to start secureHdfsFixture on Java 17 #78423

mark-vieira opened this issue Sep 28, 2021 · 13 comments
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs :Security/Security Security issues without another label Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. Team:Security Meta label for security team >test-failure Triaged test failures from CI

Comments

@mark-vieira
Copy link
Contributor

mark-vieira commented Sep 28, 2021 •
edited
Loading

Looks like this just started happening on Sep 26th. I suspect this is auto security related in some way, but don't quote me on that. Looks ot be specific to running on Java 17.

Build scan:
https://gradle-enterprise.elastic.co/s/us3iizodx5g3k/console-log?task=:x-pack:plugin:searchable-snapshots:qa:hdfs:secureHdfsFixture

Repro line:
./gradlew :x-pack:plugin:searchable-snapshots:qa:hdfs:integTestSecure -Druntime.java=17

Reproduces locally?:
Yes

Applicable branches:
master

Failure history:
https://gradle-enterprise.elastic.co/scans/failures?failures.failureClassification=all_failures&failures.failureMessage=Execution%20failed%20for%20task%20*%0A%3E%20Failed%20to%20start%20secureHdfsFixture&search.relativeStartTime=P7D&search.timeZoneId=America/Los_Angeles

Failure excerpt:

    java.io.IOException: Login failure for hdfs/[email protected] from keytab /home/mark/workspaces/elasticsearch/test/fixtures/krb5kdc-fixture/testfixtures_shared/shared/hdfs/keytabs/hdfs_hdfs.build.elastic.co.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user
    
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1066)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:286)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1081)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:376)
        at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
        at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1027)
        at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:485)
        at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:444)
        at hdfs.MiniHDFS.main(MiniHDFS.java:114)
    Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
    
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:880)
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:743)
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
        at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:734)
        at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:672)
        at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:670)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:670)
        at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:580)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1057)
        ... 9 more
    2021-09-28 16:55:47,664 INFO  [main] hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1789)) - Shutting down the Mini HDFS Cluster
    Exception in thread "main" java.io.IOException: Login failure for hdfs/[email protected] from keytab /home/mark/workspaces/elasticsearch/test/fixtures/krb5kdc-fixture/testfixtures_shared/shared/hdfs/keytabs/hdfs_hdfs.build.elastic.co.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user
    
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1066)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:286)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1081)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:376)
        at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
        at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1027)
        at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:485)
        at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:444)
        at hdfs.MiniHDFS.main(MiniHDFS.java:114)
    Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
    
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:880)
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:743)
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
        at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:734)
        at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:672)
        at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:670)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:670)
        at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:580)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1057)
        ... 9 more
@mark-vieira mark-vieira added >test-failure Triaged test failures from CI :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs :Security/Security Security issues without another label labels Sep 28, 2021
@elasticmachine elasticmachine added Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. Team:Security Meta label for security team labels Sep 28, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security (Team:Security)

@mark-vieira mark-vieira changed the title [CI] Searchable snapshots tests fail to start secureHdfsFixture [CI] Searchable snapshots tests fail to start secureHdfsFixture on Java 17 Sep 29, 2021
@masseyke
Copy link
Member

masseyke commented Oct 1, 2021

This looks suspiciously like what I was seeing on my machine. The root cause was that the keytab was using the deprecated des3-cbc-sha1-kd encryption type. I fixed it locally by switching the keytab to use aes256-cts-hmac-sha1-96 instead. But since CI wasn't failing I didn't commit this (since I wasn't sure what the FIPS implications would be). Sounds like it's time to make this change though.

@mark-vieira
Copy link
Contributor Author

But since CI wasn't failing I didn't commit this (since I wasn't sure what the FIPS implications would be). Sounds like it's time to make this change though.

Yeah, we don't run the full Java support matrix for every pull request. We generally just run with the minimum runtime Java version (which for master at the moment is Java 11).

@masseyke masseyke mentioned this issue Oct 5, 2021
Merged
@masseyke
Copy link
Member

masseyke commented Oct 5, 2021

I created #78703 with the changes I had made locally. This is outside of my area of expertise though, so glad to close that one if there's a better solution.

@ywangd
Copy link
Member

ywangd commented Oct 12, 2021

Another one: https://gradle-enterprise.elastic.co/s/64cq7tjqhnaay

@markharwood
Copy link
Contributor

Another https://gradle-enterprise.elastic.co/s/nes4gncakuwea

masseyke added a commit that referenced this issue Oct 18, 2021
@masseyke
Changing test keytab to use aes256-cts-hmac-sha1-96 instead of des3-c…
dcd2769 
…bc-sha1-kd (#78703)

The des3-cbc-sha1-kd encryption type is deprecated and no longer supported by newer jvm, causing tests
that use the krb5kdc-fixture to fail. This commit changes the encryption type of the test keytab to
aes256-cts-hmac-sha1-96.
Relates #78423
@masseyke
Copy link
Member

I've merged the PR that ought to fix this.

@Luegg
Copy link
Contributor

Luegg commented Oct 26, 2021

@masseyke looks like the error still persists: https://gradle-enterprise.elastic.co/s/orcgb5gttvgss

@masseyke
Copy link
Member

OK I can reproduce this on java 17. It works fine on Java 16. Here's what I used to reproduce it:
./gradlew :x-pack:plugin:searchable-snapshots:qa:hdfs:secureHdfsFixture
I'll look into it.

@masseyke
Copy link
Member

Looking at the keytab from that failure with ksutil, it's still using the des3-cbc-sha1-kd encryption type. There are a couple of files that still reference that. I'll have a PR up in a few minutes to fix that.

masseyke added a commit to masseyke/elasticsearch that referenced this issue Oct 26, 2021
@masseyke
Changing test keytab to use aes256-cts-hmac-sha1-96 instead of des3-c…
2f0395e 
…bc-sha1-kd (elastic#78703)

The des3-cbc-sha1-kd encryption type is deprecated and no longer supported by newer jvm, causing tests
that use the krb5kdc-fixture to fail. This commit changes the encryption type of the test keytab to
aes256-cts-hmac-sha1-96.
Relates elastic#78423
@masseyke masseyke mentioned this issue Oct 26, 2021
Merged
@masseyke
Copy link
Member

Oh actually it's just that I hadn't backported this to the 7.x line because I hadn't realized that it was going to be built with Java 17. I've just created the PR to backport the fix from master to 7.16.

masseyke added a commit that referenced this issue Oct 26, 2021
@masseyke
Changing test keytab to use aes256-cts-hmac-sha1-96 instead of des3-c…
ca97b68 
…bc-sha1-kd (#78703) (#79874)

The des3-cbc-sha1-kd encryption type is deprecated and no longer supported by newer jvm, causing tests
that use the krb5kdc-fixture to fail. This commit changes the encryption type of the test keytab to
aes256-cts-hmac-sha1-96.
Relates #78423 #78703
@tlrx
Copy link
Member

tlrx commented Nov 3, 2021

I don't see any failure since the last backport was merged so I'm closing this issue. Thanks @masseyke !

@tlrx tlrx closed this as completed Nov 3, 2021
jkakavas pushed a commit to jkakavas/elasticsearch that referenced this issue Nov 9, 2021
@masseyke @jkakavas
Changing test keytab to use aes256-cts-hmac-sha1-96 instead of des3-c…
c5e83f8 
…bc-sha1-kd (elastic#78703)

The des3-cbc-sha1-kd encryption type is deprecated and no longer supported by newer jvm, causing tests
that use the krb5kdc-fixture to fail. This commit changes the encryption type of the test keytab to
aes256-cts-hmac-sha1-96.
Relates elastic#78423
jkakavas added a commit that referenced this issue Nov 9, 2021
@jkakavas @masseyke
Changing test keytab to use aes256-cts-hmac-sha1-96 instead of des3-c…
fc047e7 
…bc-sha1-kd (#78703) (#80537)

The des3-cbc-sha1-kd encryption type is deprecated and no longer supported by newer jvm, causing tests
that use the krb5kdc-fixture to fail. This commit changes the encryption type of the test keytab to
aes256-cts-hmac-sha1-96.
Relates #78423

Co-authored-by: Keith Massey <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs :Security/Security Security issues without another label Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. Team:Security Meta label for security team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@markharwood @masseyke @tlrx @Luegg @ywangd @mark-vieira @elasticmachine