Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sun/security/pkcs11 tests failing on RHEL8 systems #2900

Open
sxa opened this issue Jan 30, 2023 · 12 comments
Open

sun/security/pkcs11 tests failing on RHEL8 systems #2900

sxa opened this issue Jan 30, 2023 · 12 comments
Assignees
@smlambert @sxa
Labels
arch:x64 critical provider:aws testFail
Milestone
2024-11 (November)

Comments

@sxa
Copy link
Member

sxa commented Jan 30, 2023

Please set the title to indicate the test name and machine name where known.

To make it easy for the infrastructure team to repeat and diagnose, please
answer the following questions:

Any other details: Split out from #2030 as that issue is resolved
See also #2360 and #1829 which are related to issues on this RHEL8 system.
@sxa sxa added the testFail label Jan 30, 2023
@sxa
Copy link
Member Author

sxa commented Jan 30, 2023 •
edited
Loading

Failures occurred on the s390x RHEL8 system too ... Will assign myself so I can discuss with the Red Hat internal team.

@sxa sxa self-assigned this Jan 30, 2023
@sxa sxa added the critical label Jan 30, 2023
@sxa sxa added this to the 2023-02 (February) milestone Feb 6, 2023
@sxa sxa changed the title sun/security/pkcs11 tests failing on test-aws-rhel8-x64-1 sun/security/pkcs11 tests failing on RHEL8 systems Feb 6, 2023
@sxa
Copy link
Member Author

sxa commented Feb 8, 2023

Upstream issue: https://bugs.openjdk.org/browse/JDK-8295343

@sxa sxa mentioned this issue Feb 21, 2023
Closed
@sxa
Copy link
Member Author

sxa commented Mar 2, 2023

@smlambert What would you propose as the next action here?

@smlambert
Copy link
Contributor

Exclude the tests in our problem lists under https://bugs.openjdk.org/browse/JDK-8295343 (and no we currently do not support by OS sub-flavours and supporting such granular exclusions is currently not in plan, unless somehow jtreg already offers that feature), and try to see an expedited fix to JDK-8295343 to be able to re-enable the tests quickly.

@sxa
Copy link
Member Author

sxa commented Mar 20, 2023 •
edited
Loading

See also https://ci.adoptium.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_2/86/testReport/ and https://ci.adoptium.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0/82/testReport/ which ran on test-equinix_esxi-ubuntu2204-x64-1 and test-equinix_esxi-ubuntu2204-x64-2, suggesting that Ubuntu 22.04 may be subject to the same issues as they both failed the following in the jdk_security3 suites:

New grinders to determine problematic tests

RHEL8/x64 failures (JDK8):

On test-docker-ubuntu2204-x64-1 (JDK20):

On test-equinix_esxi-ubuntu2204-x64-1 (JDK8):

Ubuntu 20.04: https://ci.adoptium.net/job/Grinder/6905/ (Same three failures as Ubuntu 22.04)

RHEL8/x64 JDK20: https://ci.adoptium.net/job/Grinder/6909/console - PASSED

Ubuntu 22.04 x64 JDK20: All passed: https://ci.adoptium.net/job/Grinder/6904/

(I was assuming we'd have a nice simple list of things to exclude ... :-) )

On RHEL8/x64: JDK8 (14 failures) JDK11 (16 failures) JDK17 (12 failures) JDK19 (12 failures) JDK20 Nightly (0 failures!) (Note - failures can be /2 since each one is there twice for the _0 and _1 suites)

Test JDK8 JDK11 JDK17 JDK19 JDK20
pkcs11/Signature/ByteBuffers [*] ❌ ❌ ✅ ✅ ✅
pkcs11/Signature/ReinitSignature [*] ❌ ❌ ✅ ✅ ✅
pkcs11/Signature/TestRSAKeyLength [*] ❌ ❌ ❌ ❌ n/a
pkcs11/fips/TestTLS12 ❌ ✅ n/a n/a n/a
tools/autotest.sh ❌ ❌ n/a n/a n/a
pkcs11/rsa/TestKeyFactory [+] ❌ ❌ ❌ ❌ ✅
pkcs11/rsa/TestSignatures ❌ ❌ ❌ ❌ ✅
pkcs11/KeyStore/Basic.java ✅ ❌ ❌ ❌ n/a
pkcs11/KeyPair/TestKeyPairGenerator n/a ❌ ❌ ❌ n/a
tools/NssTest n/a n/a ❌ ❌ n/a
lib/cacerts/VerifyCACerts n/a n/a n/a ❌ n/a
X509TrustManager/Distrust.java n/a n/a n/a ❌ n/a

[*] - the top 3 in this table are the ones definitely related to the enforced minimum key lengths in RHEL8, and suggests a fix may have gone into to 17+
[+] - Note that while sun/security/pkcs11/rsa/TestKeyFactory.java.TestKeyFactory is a failure, sun/security/pkcs11/rsa/TestKeyFactory passed

@sxa sxa modified the milestones: 2023-03 (March), 2023-04 (April) Mar 28, 2023
@sxa sxa modified the milestones: 2023-04 (April), 2023-05 (May) May 16, 2023
@sxa sxa modified the milestones: 2023-05 (May), 2023-06 (June) Jun 6, 2023
@sxa sxa modified the milestones: 2023-06 (June), 2023-07 (July) Jul 7, 2023
@sxa sxa modified the milestones: 2023-11 (November), Backlog Jan 3, 2024
@sxa sxa mentioned this issue Jan 3, 2024
Open
@sxa
Copy link
Member Author

sxa commented Nov 5, 2024

Need to determine correct actions in our exclude list or upstream in order to help this to pass consistently and run the appropriate tests.

@sxa sxa modified the milestones: Backlog, 2024-11 (November) Nov 5, 2024
@sxa
Copy link
Member Author

sxa commented Nov 19, 2024

@judovana Can you confirm whether you've seen these on your systems, and if you've excluded them in line with the table above?

@judovana
Copy link

I had checked all our rhel8 jobs, and the tests you are reffering are run, and passing. More details sent offlist.

@sxa
Copy link
Member Author

sxa commented Nov 20, 2024 •
edited
Loading

Some new tests (Since the ones in the table above were from about 18 months ago)

JDK OS arch suite Results
jdk8u RHEL8 x64 jdk_security3_0 1 failure sun/security/pkcs11/fips/TestTLS12.java.TestTLS12
jdk8u UBI9 x64 jdk_security3_0 7 failures (*)
jdk8u UBI9 x64 jdk_security3 Same 7 failures as above (each duplicated twice)
jdk8u UBI9 arm64 jdk_security3 1 failure - autotest.sh
jdk8u UBI8 arm64 jdk_security3 PASS
jdk8u UBI9 arm64 jdk_security3_0 1 failure keytool/autotest.sh
jdk8u CS9 arm64 jdk_security3 2 failures autotest.sh and https://ci.adoptium.net/job/Grinder/11566/testReport/javax_net_ssl_ServerName_SSLEngineExplorerMatchedSNI/java/SSLEngineExplorerMatchedSNI/ javax.net.ssl.SSLProtocolException: Input record too big: max = 16709 len = 42359. Running with 100 iterations SNI passed 100/100
jdk8u(RH) RHEL8 x64 jdk_security3 3 failures autotest.sh, standard.sh and TestTLS12
jdk8u(RH) UBI9 x64 jdk_security3 8 failures 5 Secmod , standard.sh autotest.shandLogin.sh`
jdk8u(Zulu) RHEL8 x64 jdk_security3_0 Three failures
jdk8u(Zulu) UBI9 x64 jdk_security3_0 10 failures
jdk8u(Zulu) UBI9 x64 jdk_security3_0 10 failures
jdk8u(Bish) RHEL8 x64 jdk_security3_0 5 failures
jdk8u(D'well) RHEL8 x64 jdk_security3_0 8 failures
jdk8u(Semeru) RHEL8 x64 jdk_security3_0 1 failure SNI test (failed on Temurin on CS9)
jdk8u(Semeru) UBI9 x64 jdk_security3_0 5 failures Three Secmod issues, Login.sh and autotest.sh
jdk11u RHEL8 x64 jdk_security3_0 PASS
jdk17u RHEL8 x64 jdk_security3_0 4 failures
jdk11u UBI9 x64 jdk_security3_0 PASS
jdk17u UBI9 x64 jdk_security3_0 4 failures - same as on RHEL8
jdk21u RHEL8 x64 jdk_security3_0 4 failures - same as jdk17u
jdk21u UBI9 x64 jdk_security3_0 4 failures - same as jdk17u

(*) Here are the seven failures on UBI9:
sun/security/pkcs11/Provider/Login.sh.Login 1.4 sec 1
sun/security/pkcs11/Secmod/AddPrivateKey.java.AddPrivateKey 0.27 sec 1
sun/security/pkcs11/Secmod/GetPrivateKey.java.GetPrivateKey 0.26 sec 1
sun/security/pkcs11/Secmod/JksSetPrivateKey.java.JksSetPrivateKey 0.2 sec 1
sun/security/pkcs11/Secmod/LoadKeystore.java.LoadKeystore 0.21 sec 1
sun/security/pkcs11/Secmod/TrustAnchors.java.TrustAnchors 0.23 sec 1
sun/security/tools/keytool/autotest.sh.autotest 2.4 sec 2

Noting that when trying with the Red Hat jdk8u build there was one additional failure:

Zulu showed three failures on RHEL8:

And zulu had 10 on UBI9:

JDK17u/21u failures on RHEL8/UBI9:

@sxa
Copy link
Member Author

sxa commented Nov 20, 2024 •
edited
Loading

Summary of the above (unless otherwise stated these comparisons are on jdk8u/x64)

  • The keytool/autotest failure occurs almost everywhere (except Temurin JDK8/RHEL8, although it did fail with the RH build)
  • pkcs11/fips/testTLS12 has had a bit of extra analysis in JDK8 fips/TestTLS12 failure on Ubuntu 24.04 aqa-tests#5420 (comment)
  • The aarch64 runs on UBI9 seem much more reliable than on x64 (only the autotest failure) and on UBI8 it passes cleanly
  • CentOS9 stream has one additional failure vs UBI9 on aarch64 (CS9 untested on x64) SSLEngineExplorerMatchedSNI/
  • On x64, UBI9 has several more failures, including the five sun/security/pkcs11/Secmod tests which faili with libsoftokn3 version not found, set to 0.0: /usr/lib64/libsoftokn3.so
  • The Red Hat JDK8 has one additional failure in standard.sh vs Temurin
  • The Azul builds have three additional failures in on both RHEL8 and UBI9: DHEKeySigning which gives Expected to generate ServerHello series messages of 1387 bytes, but not 1643, CacertsLimit (possibly a different cacerts bundle) and SSLExceptionForIOIssue (A timeout issue)
  • jdk11u passes the tests on UBI9 but jdk17u has four failures (may be CA bundle related? Trying a temurin without our local bundles here) ... That was jdk8. 21 is here

@sxa
Copy link
Member Author

sxa commented Nov 21, 2024

Builds without our custom CA Certs bundle for feeding to the upstream parameters in Grinder:

Running on UBI9 with the jdk21u build above at https://ci.adoptium.net/job/Grinder/11651/console (May take a while since there are a lot of jobs queued up)

@sxa
Copy link
Member Author

sxa commented Nov 22, 2024 •
edited
Loading

Running on UBI9 with the jdk21u build above at https://ci.adoptium.net/job/Grinder/11651/console (May take a while since there are a lot of jobs queued up)

Showed the same four errors so it seems unrelated our custom cacerts.

Running with Red Hat jdk21u on UBI9: https://ci.adoptium.net/job/Grinder/11717 - Failed the same four tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@smlambert smlambert

@sxa sxa

Labels
arch:x64 critical provider:aws testFail
Projects
Adoptium Backlog
Status: Todo
Milestone
2024-11 (November)
Development

No branches or pull requests
3 participants
@smlambert @judovana @sxa