Layered container image generation

[tfriman]

Description

Possibility to create layered container images when using JVM. Reasoning here is that those dependencies that change rarely are being shared as much as possible. Usually only app code is modified thus resulting a very small change. In addition this approach would allow (security) patching lower layers.

Implementation ideas

One way would be to use fast-jar directories directly:

layer0 contains jvm
layer1 contains boot-lib
layer2 contains quarkus
layer3 contains lib
layer4 contains app

https://blog.tratif.com/2020/05/20/spring-tips-2-layered-jars-with-spring-boot-2-3-0/
gives one example what this means in practise.

[geoand]

cc @maxandersen

[maxandersen]

@geoand we did this for jib, right ?

Doing I for docker or rather enable it could be to just have the new "app" output split jars into well-defined folders to allow for creating a dockerfile that can just copy each well known folder / file names in separate operations.

@stuartwdouglas that should fit with new devmode approach too?

[geoand]

@maxandersen correct, for jib we already did that

[famod]

I guess the situation has not changed lately and nobody worked on this, right?

PS: Coming from https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/Separate.20lib.20dir.20for.20project.20deps.3F

[geoand]

Not sure, but the container-image for the fast jar might already do this

[famod]

This seems to be the fast-jar template for Dockerfile:
https://github.com/quarkusio/quarkus/blob/master/devtools/platform-descriptor-json/src/main/resources/templates/dockerfile-fast-jar.ftl

Unfortunately, the fast-jar layout does not help in case you have multiple modules since those still end up in a folder (quarkus-app/lib/main) that also contains other "3rd-party" deps, including Quarkus artifacts.

/cc @stuartwdouglas

[stuartwdouglas]

Maybe we could put them in the app dir instead, but I am not sure if that is right either.

We already have this info in io.quarkus.bootstrap.model.AppModel#localProjectArtifacts so it should be a pretty simple change.

[famod]

@stuartwdouglas

Maybe we could put them in the app dir instead, but I am not sure if that is right either.

"right" in the sense of docker layering or do you mean something else?
What about introducing something like lib-app or app-lib?

[stuartwdouglas]

Right in that maybe they should be their own layer. Maybe they could go into lib/app

[famod]

What about adding a flag which moves those "localProjectArtifacts" to app, document it and see how it goes (gathering feedback from users)?

[famod]

@stuartwdouglas WDYT about ^^^ ?

[lburgazzoli]

bumping this issue.

+1 to have the localProjectArtifacts moved to app or lib/app
(having a way to create/configure additional layer would be nice too)

[geoand]

@lburgazzoli you mean when using container-image-docker? Asking because container-image-jib already does this

[lburgazzoli]

@lburgazzoli you mean when using container-image-docker? Asking because container-image-jib already does this

No, I'm talking about container-image-jib.
From which version it has been implemented ? Asking because in my project with quarkus 2.7.5 I have:

.
├── app
│  └── cos-connector-mongodb-0.1-999-SNAPSHOT.jar
├── lib
│  ├── boot
│  │  ├── io.quarkus.quarkus-bootstrap-runner-2.7.5.Final.jar
│  │  ├── io.quarkus.quarkus-development-mode-spi-2.7.5.Final.jar
│  │  ├── io.smallrye.common.smallrye-common-io-1.10.0.jar
│  │  ├── org.glassfish.jakarta.json-1.1.6.jar
│  │  ├── org.graalvm.sdk.graal-sdk-21.3.1.jar
│  │  ├── org.jboss.logging.jboss-logging-3.4.3.Final.jar
│  │  ├── org.jboss.logmanager.jboss-logmanager-embedded-1.0.9.jar
│  │  └── org.wildfly.common.wildfly-common-1.5.4.Final-format-001.jar
│  └── main
│     ├── ...
│     ├── org.bf2.cos-connector-action-jsonpath-999-SNAPSHOT.jar
│     ├── org.bf2.cos-connector-action-serdes-999-SNAPSHOT.jar
│     ├── org.bf2.cos-connector-camel-999-SNAPSHOT.jar
│     ├── org.bf2.cos-connector-camel-component-routecontroller-0.0.1-SNAPSHOT.jar
│     ├── org.bf2.cos-connector-kafka-999-SNAPSHOT.jar
│     ├── org.bf2.cos-connector-serdes-999-SNAPSHOT.jar
│     ├── org.bf2.cos-fleet-catalog-kamelets-999-SNAPSHOT.jar
│     └── ...
├── quarkus
│  ├── generated-bytecode.jar
│  ├── quarkus-application.dat
│  └── transformed-bytecode.jar
├── quarkus-app-dependencies.txt
└── quarkus-run.jar

Where the org.bf2dependencies are local but mixed with other dependencies

[geoand]

Oh, now I see what you mean.

So you are essentially asking for the dependencies that are part of the project to be in their own layer (I assume above the others since it changes faster)?

[lburgazzoli]

Oh, now I see what you mean.

So you are essentially asking for the dependencies that are part of the project to be in their own layer (I assume above the others since it changes faster)?

Yes correct and because of smallrye/jandex#213, all the local dependency change for each build even if nothing has changed which makes the lib/main layer less effective

[geoand]

I see. Do you perhaps have any samples we could use to test the feature if/when it's developed?

[lburgazzoli]

This repo is a good candidate

[geoand]

Thanks! I try and have a look for 2.11

[geoand]

Is there a Quarkus application in any of those modules? I don't see any

[geoand]

@aloubyansky in a multi-module project, is there a way to get the in-project dependencies of the Quarkus application from the application model?
I should clarify that I would need this in during the regular Quarkus build, not in dev mode.

[aloubyansky]

@geoand ApplicationModel.getDependencies().stream().filter(Dependency::isWorkspaceModule).collect(Collectors.toList()) should do that.

[aloubyansky]

Actually, that might not be available during the build, given that we don't perform workspace discovery for LaunchMode.NORMAL.

[aloubyansky]

In case we are adding a config option for the new layout, we could use that to enable workspace discovery for the ApplicationModel resolution.

[geoand]

I think that's a good idea

[geoand]

@aloubyansky I started looking into what is needed for this and the way I could get it to work (without yet involving a config option) is with these changes: geoand@f808994

Now I am sure those are too invansive and have implications that I don't understand, so if you have suggestions on how to proceed (or if you want to take this up yourself), this would be very welcome :)

[aloubyansky]

Sure, enabling workspace discovery for all modes would make local dependencies info always available. The cost would be extra time necessary to load the workspace (all the POMs, etc). Which isn't that much.

How were you thinking of implementing the rest? Changing the default lib layout or adding a config option for it? Just to clarify whether we are ending up with a config option anyway.

In this case, consulting a config option that is configured in an application.properties, application.yaml or another config source before the dependencies are resolved is problematic. It's a chicken&egg problem. We already have hacks like https://github.com/quarkusio/quarkus/blob/main/independent-projects/bootstrap/core/src/main/java/io/quarkus/bootstrap/app/ConfiguredClassLoading.java#L64 where we checks classloading config but only in the application.properties. So, that could be another point in favor of enabling workspace discovery in all modes.

[geoand]

How were you thinking of implementing the rest? Changing the default lib layout or adding a config option for it? Just to clarify whether we are ending up with a config option anyway.

The contaimer image layering part is easy. I already have code that creates a layer for "fast changing" dependencies, so I just need a way to identify them - in this case that would mean I can figure out which of the dependencies are local project dependencies. No changes to the jar layout itself are needed.

In this case, consulting a config option that is configured in an application.properties, application.yaml or another config source before the dependencies are resolved is problematic. It's a chicken&egg problem.

Yeah, I figured that would be the case. To be honest, I would be fine if this were only in application.properties or even only in build system properties. We can make that very clear in the documentation of the feature.

[geoand]

The second commit in https://github.com/geoand/quarkus/tree/%239818 shows how this would be used. I tried it and it works as expected

[aloubyansky]

Ok, let's start with a build system property and system property then. I can open a PR for it this afternoon.

[geoand]

Awesome, thanks!

[geoand]

#25941 makes use of the capability added by @aloubyansky to Quarkus bootstrap to support using a separate container image layer for local project dependencies.

I should note that the file system structure is not affected, so one won't see the change reflected there. The change can be seen however using dive on the generated image.

[famod]

Very nice, from:

to:

[geoand]

Super!