Unable to find valid certification path

[nboissel]

This issue appears on Docker official image (tried with AdoptOpenJDK 8 and 11 in hostpot/amd64 flavors) but I think it can be reproduced on any AdoptOpenJDK Hotspot.

I get the usual PKIX error each time I request some APIs
Error : javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
It wouldn't be a problem if this was an auto-signed certificate but it appears on all french government open APIs I tested (for example https://entreprise.data.gouv.fr/api/sirene/v1/siret/).
Those certificates are signed by the same root authority CN = Certigna, O = Dhimyotis, C = FR which seems legit.
It works on OpenJDK so I imagine it is "just" a difference in root certificate list.

Was this root certificate voluntarily invalidated? Is it just an omission ?
I know I can add this certificate manually using keystore but I was wondering if this should not be integrated by default.

[sxa]

@nboissel Does it also fail with the latest nightly build from https://adoptopenjdk.net/nightly.html ?

[nboissel]

I just tried with the last build OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+3-201912032132) and I still got the same error.

[ggam]

I'm facing the same issue on JDK 13:

openjdk version "13" 2019-09-17
OpenJDK Runtime Environment AdoptOpenJDK (build 13+33)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13+33, mixed mode, sharing)

[ggam]

Just found in my case it was my fault. OpenLiberty doesn't use JDK cacerts by default. I needed to add the following to the server.xml file:

<ssl id="defaultSSLConfig" trustStoreRef="myTrustStore"/>

<keyStore id="myTrustStore" location="${java.home}/lib/security/cacerts" password="changeit" />

[tresf]

Is there a chance that it's due to the SHA-1 signature in the certificate chain per https://openjdk.java.net/jeps/288?

Quoting:

Improve the security configuration of the JDK by providing a more flexible mechanism to disable X.509 certificate chains with SHA-1 based signatures.

Per https://www.sslshopper.com/ssl-checker.html?hostname=+https%3A%2F%2Fentreprise.data.gouv.fr:

[aahlenst]

Closing due to inactivity.

[nboissel]

I can imagine this issue does not concerns a lot of users but if there is this behavior with those root certificates, it can append for others.

I looked into the lead proposed by @tresf but I reproduce the same issue with api.insee.fr (a sub-site of data.gouv.fr) with the following chain certificate:

This is for the new version of the same service with URL https://api.insee.fr/entreprises/sirene/V3/siret/{siret}

[tresf]

Some conversation over here reads very similar: adoptium/temurin-build#676, but looking at the cacerts file, where should this be?

For example, if I cd to JAVA_HOME/lib/security and type this command, I get quite a few certs...

echo "changeit" | keytool -list -v -keystore cacerts |grep -i VeriSign

... however when I try the same command with Certigna, I get nothing

echo "changeit" | keytool -list -v -keystore cacerts |grep -i Certigna

... yet Chrome trusts Certigna as a valid, Root Certificate Authority.

[tresf]

Related: habitat-sh/core-plans#1488 (comment).

Quoting:

I'm currently facing a case where I need to access an url validated by a root certificate Certigna not included in the JRE but trusted by Chrome and Firefox.

[aahlenst]

The cacert file bundled with OpenJDK does not match the list of accepted CAs that is maintained by Mozilla. Historically, when only Oracle was in charge, they were significantly more forgiving than the browser vendors if I remember correctly.

Personally, I hate the fact that our cacert does not match Mozilla's. I think there was a discussion on our Slack but Martijn wanted to discuss that privately because he considered it to be a sensitive security issue (I don't). Hence I don't know what our thinking is here.

As a side note, there's adoptium/installer#105. The gist is: At least when you install the Linux packages, Java should either use the system's certificate store or get a cacert generated from Mozilla's. I didn't get around implementing the necessary tooling in Java, yet. If one of you wants to help out, please get in touch.

[tresf]

I've worked with Firefox's certdb format a bit, but I've found the command-line tools required for interacting with it aren't available installed by default with certain Linux distributions.

Whereas you're not happy with the fact that cacert differs from Mozilla's, I've grown equally frustrated with Mozilla shipping it's own as it causes similar issues to this, but from a browser perspective (fortunately Mozilla recently allowed enterprise policies to use the system's store, but that's a completely different subject).

Anyway, I vaguely remember a property for Windows called -Djavax.net.ssl.trustStoreType=WINDOWS-ROOT, is there such a flag for Linux? Linux seems to handle CA Certs a bit differently than Windows and Mac, so I'm not sure what -- if any -- would be a sane default. I guess importing them at install time would suffice, but that script would need to re-run on each update (or better, any time the Firefox store is updated), no?

[tresf]

he considered it to be a sensitive security issue (I don't)

When 20,000 certificates are revoked, it's public knowledge these days, and issuance is as quick as running certbot on a host, I tend to agree. These things should be in the open, especially when CAs trusted by major browsers are being rejected by a default AdoptOpenJDK install.

Of course, the most secure system is one which trusts no CAs at all, but that would be a difficult selling point. :D

[aahlenst]

To verify whether the cacert file is really the culprit here:

1. Build a bundle with https://raw.githubusercontent.com/curl/curl/master/lib/mk-ca-bundle.pl.
2. Create the cacert with https://github.com/alastairmccormack/keyutil.
3. Replace the bundled one with the generated file.
4. Test.

If someone could do that, I‘d be very grateful.

@tresf Debian has tooling to create a cacert file from the system‘s keystore. There is a hook system that updates the cacert every time the system‘s keystore is changed. lib/security/cacert is actually a symlink to that file. My plan is to create Java-based tools that replicate that on Red Hat and friends as well as all Debian derivatives. So you have an up-to-date CA list according to your needs that matches the behavour of your system. In case of Debian, it‘s identical with Mozilla‘s list.

[tresf]

@aahlenst Using the perl script, I was able to create two cert files:

• certdata.txt
• ca-bundle.crt

Next, I snagged keyutil.

Last, I attempted to create my own cacert bundle, but the file it's creating is empty.

Here's my commands:

# install curl on Ubuntu, mk-ca-bundle requires it
sudo apt install curl
# get mk-ca-bundle.pl
wget https://raw.githubusercontent.com/curl/curl/master/lib/mk-ca-bundle.pl
chmod +x mk-ca-bundle.pl
./mk-ca-bundle.pl

# get keyutil
wget https://github.com/use-sparingly/keyutil/releases/download/0.4.0/keyutil-0.4.0.jar
# create a new cacert file
java -jar keyutil-0.4.0.jar --new-keystore cacert --password changeit --import-pem-file ca-bundle.crt
# validate file before deploying
echo "changeit" |keytool -list -keystore cacert

Unfortunately, I get the following:

  Enter keystore password:  Keystore type: JKS
  Keystore provider: SUN

- Your keystore contains 0 entries

I assume I'm doing something very basically wrong.

P.S. I can confirm that on Ubuntu 20.04:

• ✅ Firefox loads https://api.insee.fr/catalogue/ just fine
• ✅ openjdk11 (from Canonical repo) loads https://api.insee.fr/catalogue/ just fine.
• 🚫 AdoptOpenJDK11 fails to load https://api.insee.fr/catalogue/

I'll be able to confirm whether or not cacert override works once I can get the above working.

[aahlenst]

tl;dr: As suspected, the problem is the cacerts shipped with OpenJDK and also AdoptOpenJDK. There are significant less CA certificates in both cacerts keystores. Replacing the bundled cacerts with one generated from Mozilla's list of accepted CAs makes the problem go away.

And now, the details…

OpenJDK version:

java -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7)
OpenJDK 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)

AdoptOpenJDK version:

openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)

Sample project to reproduce: https://github.com/aahlenst/cacert-test (JDK 11 and higher). Run ./gradlew check. Two tests are in it, both should pass.

Number of bundled CA certificates with (Adopt)OpenJDK:

keytool -list -keystore jdk-14.0.1/lib/security/cacerts
Warning: use -cacerts option to access cacerts keystore
Enter keystore password:  
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 93 entries

Number of CA certificates in Mozilla's list:

keytool -list -keystore cacerts-mozilla 
Enter keystore password:  
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 138 entries

How I generated cacerts-mozilla (mk-ca-bundle.pl, keyutil-0.4.0.jar):

$ wget https://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt .
$ perl mk-ca-bundle.pl -n > ca-bundle.crt
$ java -jar keyutil-0.4.0.jar --import --new-keystore cacerts-mozilla --password changeit --force-new-overwrite --import-pem-file ca-bundle.crt

With the bundled cacerts:

./gradlew clean check

> Task :test FAILED

SampleClientTest > callSiteWithKnownCa() PASSED

SampleClientTest > callSiteWithUnknownCa() FAILED
    javax.net.ssl.SSLHandshakeException at SampleClientTest.java:16
        Caused by: javax.net.ssl.SSLHandshakeException at Alert.java:131
            Caused by: sun.security.validator.ValidatorException at PKIXValidator.java:439
                Caused by: sun.security.provider.certpath.SunCertPathBuilderException at SunCertPathBuilder.java:141

2 tests completed, 1 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///root/cacert-test/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s
4 actionable tasks: 4 executed

After replacing the bundled cacerts with the newly generated one (cp cacerts-mozilla jdk-14.0.1/lib/security/cacerts):

./gradlew clean check

> Task :test

SampleClientTest > callSiteWithKnownCa() PASSED

SampleClientTest > callSiteWithUnknownCa() PASSED

BUILD SUCCESSFUL in 2s
4 actionable tasks: 4 executed

The sites I used to test: https://google.com/ and https://api.insee.fr/catalogue/.

It's expected that JDK's shipped with Ubuntu/Debian work because they use a different cacerts build from Mozilla's list.

Side note: I tested with BellSoft Liberica 13, too. Same problem.

Personal commentary: I can see how one can argue that it makes sense to ship the same cacerts file as upstream because one of AdoptOpenJDKs objectives is to match OpenJDK as closely as possible. In this case, I think we're doing our users a disservice. We should either delegate what CA certificates are accepted to the operating system or, if this isn't possible, at least ship the best curated list of trusted CA certificates and that's Mozilla's.

[aahlenst]

@karianna Please advise how to proceed here. I see three options:

1. Try to persuade OpenJDK to follow Mozilla's lead.
2. Regenerate cacerts at AdoptOpenJDK from Mozilla's list, either on each build or before each release. Might need TSC discussion.
3. Do nothing.

Necessary information to reproduce should be present.

[karianna]

TSC has agreed in principle to add the Mozilla certs - @gdams will take a first look at it.

[breun]

I ran into this issue today as well. The Dutch government runs its own root CA, but this is not included in AdoptOpenJDK's cacerts, while it is included in the cacerts file from the ca-certificates package on CentOS 8 (which contains the CA certificates chosen by the Mozilla Foundation):

$ echo changeit | keytool -list -v -keystore $(find $JAVA_HOME -name cacerts) 2> /dev/null | grep ^Owner | grep Staat
$ echo changeit | keytool -list -v -keystore /etc/pki/ca-trust/extracted/java/cacerts 2> /dev/null  | grep ^Owner | grep Staat
Owner: CN=Staat der Nederlanden EV Root CA, O=Staat der Nederlanden, C=NL
Owner: CN=Staat der Nederlanden Root CA - G2, O=Staat der Nederlanden, C=NL
Owner: CN=Staat der Nederlanden Root CA - G3, O=Staat der Nederlanden, C=NL

I'm tempted to just symlink AdoptOpenJDK's cacerts to /etc/pki/ca-trust/extracted/java/cacerts. But it's good to hear that AdoptOpenJDK plans on adopting Mozilla's list.

[breun]

I also tried Azul Zulu Community, and that OpenJDK distribution does contain 'Staat der Nederlanden Root CA - G3':

$ echo changeit | keytool -list -v -keystore $(find $JAVA_HOME -name cacerts) 2> /dev/null | grep ^Owner | grep Staat
Owner: CN=Staat der Nederlanden Root CA - G3, O=Staat der Nederlanden, C=NL
Owner: CN=Staat der Nederlanden EV Root CA, O=Staat der Nederlanden, C=NL

Maybe they also use Mozilla's list? Although they don't have 'Staat der Nederlanden Root CA - G2'.

[aahlenst]

Sorry it took us so long. 15 should be the first one with a fresh cacerts file. 8 and 11 should get it in October. The nightly builds (every version) that appear in the next hours on the website should already have it. Please give them a whirl.