Support CA Certificates with native images

[scottfrederick]

When configuring a JDK or JRE, this buildpack will add any certificates provided by the CA Certificates buildpack to the JVM truststore. It does not appear that certificates provided by CA Certificates will be configured correctly when building a native image. Can support for CA Certificates in native images be added?

[dmikusa]

@scottfrederick Hi, we merged paketo-buildpacks/java-native-image#99 recently, which sounds like it addresses your request here. I just wanted to confirm though, as this is opened up under the Graalvm buildpack. Just want to make sure I'm not overlooking somewhere else where we need to add ca-certificates back. Thanks

[matthyx]

Hi, this request is about adding the certificates to the embedded jks keystore that is added to the executable at build time.
It is required because substrate VM doesn't support jks loading at runtime.
I have an open PR for graalVM regarding that, but in the meantime we need a way to include our internal certificates.

[matthyx]

I was going to check if I could propose a PR here, but if you're already familiar with buildpacks please go ahead (I'm a complete n00b here).

[matthyx]

FTR here is the graalVM PR: oracle/graal#3091

[scottfrederick]

As @matthyx said, there's more to do here for GraalVM in the native image case. Good documentation on this is hard to find, but there's a good hint in the GraalVM javadocs.

To fully support CA Certificates in this buildpack with native-image builds, I think what needs to happen is:

• Certificates are read from the location the ca-certificates buildpack stores them in, converted to a Java keystore format, and stored in the image (possibly using libjvm/certificate_loader.go).
• The system property -Djavax.net.ssl.trustStore is set to the path of the generated Java keystore file at native image compile time.

[matthyx]

100% correct
So who's implementing it?
I'll badly need it once Spring Boot reaches 2.5.0 stable :-)

[dmikusa]

OK, thanks for the clarifications. Let me take a look at this tomorrow and see what we can do.

[matthyx]

Hi @dmikusa-pivotal any news?

[dmikusa]

Well, the good news is that I think we are doing similar things in other buildpacks so it shouldn't be too hard to add this in here. It'll take some work but I have a couple of other priorities in my queue first, so I'd guess maybe 1-2 weeks for me to implement. If you are interested in getting it done more quickly and could send a PR, I can definitely make time to review/merge that in.

[matthyx]

Thanks @dmikusa-pivotal I think it's fine for a timeline. You'll probably make it better and faster than me...

[dmikusa]

Certificates are read from the location the ca-certificates buildpack stores them in, converted to a Java keystore format, and stored in the image (possibly using libjvm/certificate_loader.go).
The system property -Djavax.net.ssl.trustStore is set to the path of the generated Java keystore file at native image compile time.

Digging some more, GraalVM buildpack is already running the code that should be importing container certs (and as far as I can tell, also ca-certs provided certs, into the JVM's default cacerts. The GraalVM buildpack is loading these into the keystore here.

My suspicion was that you could make this work right now by setting $BP_NATIVE_IMAGE_BUILD_ARGUMENTS='-Djavax.net.ssl.trustStore=$JAVA_HOME/jre/lib/security/cacerts' (I'm not 100% sure it'll interpret JAVA_HOME correctly, it might require the full path to the layer). I tried this a few ways and I can see it passing the -Djavax.net.ssl.trustStore argument into the command to build the image. That said, my resulting image doesn't work. It's giving me trust errors when I try to make an HTTPS connections.

I can see ca-certs buildpack running and importing my custom CA into the image. There is unfortunately no logging when GraalVM buildpack loads the container certs into the JVM default cert. I need to confirm this is happening as I expect. Failing that, I will have to poke at the native image build command a bit more. I was hoping to get this working before I start hacking at the code. It seems like the changes should be minimal, I just wanted to figure out specifically what needs to change.

[matthyx]

Awesome news @dmikusa-pivotal I will try to find some time today to try out the trick with $BP_NATIVE_IMAGE_BUILD_ARGUMENTS and let you know!

[dmikusa]

My suspicion was that you could make this work right now by setting $BP_NATIVE_IMAGE_BUILD_ARGUMENTS='-Djavax.net.ssl.trustStore=$JAVA_HOME/jre/lib/security/cacerts' (I'm not 100% sure it'll interpret JAVA_HOME correctly, it might require the full path to the layer). I tried this a few ways and I can see it passing the -Djavax.net.ssl.trustStore argument into the command to build the image. That said, my resulting image doesn't work. It's giving me trust errors when I try to make an HTTPS connections.

OK, so I figured out what I was messing up yesterday. When I was merging updates recently, I messed up the order and the ca-certs buildpack was not running first. Thus GraalVM was running, importing the default set of certs, then ca-certs was running and adding my custom cert, but it was already too late because GraalVM had run and generated the cacerts file with the container certs. Anyway, after fixing this. It's working for me.

So I set BP_NATIVE_IMAGE_BUILD_ARGUMENTS='--enable-https -Djavax.net.ssl.trustStore=$JAVA_HOME/jre/lib/security/cacerts'. The first argument is unrelated but required if you want HTTPS to work in a native image. The second points native-image to the correct cacerts file. It does appear that $JAVA_HOME is honored here, assuming you properly shell-escape it so that it's not evaluated in your local shell.

You should be able to use this now (although you'll need a custom java-native-image buildpack image at the moment since we haven't cut a release since adding ca-certs back into the buildpack).

What I will do is automate this and have the native-image buildpack pass in this argument conditionally, based on the presence of the ca-certs binding.

[dmikusa]

The system property -Djavax.net.ssl.trustStore is set to the path of the generated Java keystore file at native image compile time.

I'm also wondering if this is really necessary. The GraalVM buildpack is loading the ca-certificates provided certs into the default truststore for the JVM, i.e. $JAVA_HOME/jre/lib/security/cacerts. I did a quick test without the -Djavax.net.ssl.trustStore argument and it seemed to pick things up just fine. My image was able to connect a server that was using a certificate signed by the bundled CA cert.

@matthyx If you could try it without the argument too. Just make sure that you see these three key items in your build output.

1. CA Certificates Buildpack runs before GraalVM buildpack
2. CA Certificates should output Added 1 additional CA certificate(s) to system truststore, or the number of certs you've bound.
3. GraalVM should say Adding 130 container CA certificates to JVM truststore where 130 is the default of 129 certs + the number of certs you've bound in. I bound in one, so 129 + 1 == 130.

[builder] Paketo CA Certificates Buildpack 2.2.0 
[builder]   https://github.com/paketo-buildpacks/ca-certificates
[builder]   Launch Helper: Contributing to layer
[builder]     Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
[builder]   CA Certificates: Contributing to layer
[builder]     Added 1 additional CA certificate(s) to system truststore
[builder]     Writing env.build/SSL_CERT_DIR.append
[builder]     Writing env.build/SSL_CERT_DIR.delim
[builder]     Writing env.build/SSL_CERT_FILE.default
[builder]
[builder] Paketo GraalVM Buildpack 6.1.0
[builder]   https://github.com/paketo-buildpacks/graalvm
[builder]   Build Configuration:
[builder]     $BP_JVM_VERSION              11              the Java version
[builder]   Launch Configuration:
[builder]     $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
[builder]     $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
[builder]     $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
[builder]     $JAVA_TOOL_OPTIONS                           the JVM launch flags
[builder]   GraalVM JDK 11.0.11: Contributing to layer
[builder]     Downloading from https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-21.1.0/graalvm-ce-java11-linux-amd64-21.1.0.tar.gz
[builder]     Verifying checksum
[builder]     Expanding to /layers/paketo-buildpacks_graalvm/jdk
[builder]     Adding 130 container CA certificates to JVM truststore
[builder]   GraalVM Native Image Substrate VM 11.0.11
[builder]     Downloading from https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-21.1.0/native-image-installable-svm-java11-linux-amd64-21.1.0.jar
[builder]     Verifying checksum
[builder]     Installing substrate VM
[builder] Processing Component archive: /tmp/1ab725616fede21ad5ae900d00ec6d45753db6f6f4fe51a836d538c79d79614a/native-image-installable-svm-java11-linux-amd64-21.1.0.jar
[builder] Installing new component: Native Image (org.graalvm.native-image, version 21.1.0)
[builder]     Writing env.build/JAVA_HOME.override
[builder]     Writing env.build/JDK_HOME.override
...

In my case, when these three happened, native-image just did the right thing and pull in my custom cacerts.

[matthyx]

@dmikusa-pivotal I have tried to build a new version of the native-image buildpack and then substitute it on my docker host.

However I cannot test if that works, because paketo-buildpacks/graalvm:6.0.0 doesn't take my binding into account for the certificates and therefore fails to download the tar.gz for graalvm-ce-java11 because of our internal SSL introspection.

[matthyx]

Here is the error:

[INFO] Building image 'docker.io/library/job-launcher:0.0.8-SNAPSHOT'
[INFO]
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 100%
[INFO]  > Pulled builder image 'paketobuildpacks/builder@sha256:a5643f60ae81cd452935b213e1aae3042a61fe6d27ecbfb3f479b0888231c6b4'
[INFO]  > Pulling run image 'docker.io/paketobuildpacks/run:tiny-cnb' 100%
[INFO]  > Pulled run image 'paketobuildpacks/run@sha256:57768a9ede0bed24b6c176a673e2cc825c65ac319be0baa32661eb4db722dba3'
[INFO]  > Executing lifecycle version v0.11.1
[INFO]  > Using build cache volume 'pack-cache-66f994727480.build'
[INFO]
[INFO]  > Running creator
[INFO]     [creator]     ===> DETECTING
[INFO]     [creator]     4 of 11 buildpacks participating
[INFO]     [creator]     paketo-buildpacks/graalvm        6.0.0
[INFO]     [creator]     paketo-buildpacks/executable-jar 5.0.0
[INFO]     [creator]     paketo-buildpacks/spring-boot    4.2.0
[INFO]     [creator]     paketo-buildpacks/native-image   4.0.0
[INFO]     [creator]     ===> ANALYZING
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:helper" from app image
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:spring-cloud-bindings" from app image
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:web-application-type" from app image
[INFO]     [creator]     ===> RESTORING
[INFO]     [creator]     ===> BUILDING
[INFO]     [creator]
[INFO]     [creator]     Paketo GraalVM Buildpack 6.0.0
[INFO]     [creator]       https://github.com/paketo-buildpacks/graalvm
[INFO]     [creator]       Build Configuration:
[INFO]     [creator]         $BP_JVM_VERSION              11.*            the Java version
[INFO]     [creator]       Launch Configuration:
[INFO]     [creator]         $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
[INFO]     [creator]         $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
[INFO]     [creator]         $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
[INFO]     [creator]         $JAVA_TOOL_OPTIONS                           the JVM launch flags
[INFO]     [creator]       GraalVM JDK 11.0.10: Contributing to layer
[INFO]     [creator]         Downloading from https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-21.0.0.2/graalvm-ce-java11-linux-amd64-21.0.0.2.tar.gz
[INFO]     [creator]     unable to invoke layer creator
[INFO]     [creator]     unable to get dependency jdk
[INFO]     [creator]     unable to download https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-21.0.0.2/graalvm-ce-java11-linux-amd64-21.0.0.2.tar.gz
[INFO]     [creator]     unable to request https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-21.0.0.2/graalvm-ce-java11-linux-amd64-21.0.0.2.tar.gz
[INFO]     [creator]     Get "https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-21.0.0.2/graalvm-ce-java11-linux-amd64-21.0.0.2.tar.gz": x509: certificate signed by unknown authority
[INFO]     [creator]     ERROR: failed to build: exit status 1

[dmikusa]

Sorry, let's see if we can make this easier to test.

1. Are you using Spring Boot tools here? In Boot 2.4, there is no support for bindings. That is new in 2.5. If you're using one of the RC's, you can set bindings https://docs.spring.io/spring-boot/docs/2.5.0-RC1/maven-plugin/reference/htmlsingle/#build-image-customization. Also, change the builder to use dmikusa/tiny-java-native, which is a testing branch I put up to make this easier.

2. Alternatively, can you try this?

pack build <image-name> --path target/your-file.jar -e BP_NATIVE_IMAGE=true -e BP_NATIVE_IMAGE_BUILD_ARGUMENTS='--enable-https' -B dmikusa/tiny-java-native --volume "/path/to/binding/:/platform/bindings/ca-certificates"

1. BP_NATIVE_IMAGE_BUILD_ARGUMENTS and --enable-https is optional, I included that because I was making HTTPS requests as a way to validate.

2. dmikusa/tiny-java-native is a temporary copy of the tiny builder where I have the main branch of the java-native-image included so that ca-certificates is included and in the correct order. This will go away, when I cut the next java-native-image release & our builders update.

3. /path/to/binding should be on your local system pointing to a directory that looks roughly like this:

binding/
├── cert-1.crt
└── type

The contents of the type file should be ca-certificates. The other files are just the certs to include. There's one in my example, but you should be able to include more.

Let me know how that goes.

[matthyx]

Perfect, it works!

[INFO]     [creator]       https://github.com/paketo-buildpacks/ca-certificates
[INFO]     [creator]       Launch Helper: Contributing to layer
[INFO]     [creator]         Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
[INFO]     [creator]       CA Certificates: Contributing to layer
[INFO]     [creator]         Added 1 additional CA certificate(s) to system truststore
[INFO]     [creator]         Writing env.build/SSL_CERT_DIR.append
[INFO]     [creator]         Writing env.build/SSL_CERT_DIR.delim
[INFO]     [creator]         Writing env.build/SSL_CERT_FILE.default

And even builds 'till the end:

[INFO]     [creator]     Adding layer 'paketo-buildpacks/ca-certificates:helper'
[INFO]     [creator]     Adding 1/1 app layer(s)
[INFO]     [creator]     Reusing layer 'launcher'
[INFO]     [creator]     Adding layer 'config'
[INFO]     [creator]     Adding layer 'process-types'
[INFO]     [creator]     Adding label 'io.buildpacks.lifecycle.metadata'
[INFO]     [creator]     Adding label 'io.buildpacks.build.metadata'
[INFO]     [creator]     Adding label 'io.buildpacks.project.metadata'
[INFO]     [creator]     Adding label 'org.opencontainers.image.title'
[INFO]     [creator]     Adding label 'org.opencontainers.image.version'
[INFO]     [creator]     Adding label 'org.springframework.boot.spring-configuration-metadata.json'
[INFO]     [creator]     Adding label 'org.springframework.boot.version'
[INFO]     [creator]     Setting default process type 'web'
[INFO]     [creator]     Saving docker.io/library/job-launcher:0.0.8-SNAPSHOT...
[INFO]     [creator]     *** Images (575017bf258d):
[INFO]     [creator]           docker.io/library/job-launcher:0.0.8-SNAPSHOT
[INFO]     [creator]     Adding cache layer 'paketo-buildpacks/graalvm:jdk'
[INFO]     [creator]     Adding cache layer 'paketo-buildpacks/native-image:native-image'
[INFO]
[INFO] Successfully built image 'docker.io/library/job-launcher:0.0.8-SNAPSHOT'

Thank you so much for your support, now we're waiting for the 2.5.0 release.

[matthyx]

I was already using 2.5.0-M1 and bindings, but they only worked for jvm builds.
Now with your custom image native builds work behind our proxies, and they include the proper rootCA, thank you one more time!

[dmikusa]

Fantastic! Thanks @matthyx for helping me test and work through this.

Since I didn't have to change any code here, there's no new GraalVM version to cut. We do need to cut a new release of java-native-image though, to fix the ordering issue. That is presently held up for compatibility reasons, as I mentioned here -> paketo-buildpacks/java-native-image#99 (comment). In the meantime, feel free to use that temporary builder I pushed. I'll leave it published. Just make sure you switch once we cut the new release.

I'll keep this issue open until we're able to cut the new release so you can get notifications.

[sdeleuze]

@matthyx Could you please share a sample application to help me writing proper documentation (tracked by spring-attic/spring-native#689)?

[matthyx]

Sure @sdeleuze let me fork the sample apps repo and make the modifications there... I cannot share our company apps on github.

[matthyx]

Please see my modifications in the webclient sample app:
matthyx/spring-native@9daa41f

You have to build it using:
mvn spring-boot:build-image

[ajurge]

I have just tested this with springBootVersion = '2.5.0' and it only works if i convert the certificates to the PEM format using the following command: openssl x509 -in certificate.crt -inform DER -out certificate.pem -outform PEM.

 bootBuildImage {
    bindings = [
        "${project.rootDir}/bindings/ca-certificates:/platform/bindings/ca-certificates"
    ]
} 

If the certificates are in the CRT format, then i get the following error:

[creator]     failed to decode certificate from file at path "/platform/bindings/ca-certificates/certificate.crt"
[creator]     failed find PEM data
[creator]     ERROR: failed to build: exit status 1

Is this expected and we should always convert to PEM ?

[matthyx]

I think so: https://github.com/paketo-buildpacks/ca-certificates
CA certificate to trust. Should contain exactly one PEM encoded certificate.

[ajurge]

"${project.rootDir}/bindings/ca-certificates can contain multiple CA certs, but they all have to be PEM.

I have just checked /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts in the CNB image and it contains 2 CA certs that i have added to the directory above:

[dmikusa]

+1 they have to be PEM encoded, that's a requirement of ca-certificates. If that's a challenge, or you think we should permit other formats, you could open an issue against ca-certificates.

[ajurge]

I think this is fine from my point of view, we can convert them to be PEM encoded.

[dmikusa]

Sorry for the delay. This work was released and the latest versions of Java Native Image buildpack should support CA certificates.