Working group: Class loading in Quarkus

[dmlloyd]

Greetings fellow Quarkoids!

I'd like to propose the formation of a working group in order to address some long-standing class loading architectural issues within Quarkus.

Current Context and Problem Statement

At the time of this writing, the Quarkus codebase contains between 15 and 20 class loader implementations. These implementations serve a variety of purposes:

• Exposing hidden API methods
• Setting up a (different) execution environment for a variety of application launch modes
• Working around problems in upstream projects
• Customizing parent delegation in a variety of situations

Each of these class loaders has a different behavior; some are outright incorrect. Additionally, we have a number of problems relating to specific implementations. These problems cover such areas as concurrency, virtual threads, delegation problems, and more.

The ad-hoc approach to class loading served us to get the Quarkus project off the ground quickly. However, at this stage of project evolution, I think it would be a good idea to take a top-down look at what our class loading needs are, and reevaluate the architecture. I think it would be particularly wise to consider the past (what did or did not work well for WildFly), the present (what does or does not work well in our existing implementations, what are GraalVM's needs), and the future (supporting JPMS, FFM, Leyden, and other future considerations).

Proposal

I'd propose we approach this in phases. The first phase (which is already underway) would involve reducing the surface area of the problem by looking through the various existing class loader implementations and cleaning them up, documenting them, and/or removing them from the code base if they are no longer needed. See #43726 for a top-level issue covering this work.

The second phase would be a top-down analysis of how class loading should ideally work at both build and run time. This is where a more formal working group would be of most use. In this phase, we would consider what performance and behavioral characteristics are expected at both build time and run time.

A non-exhaustive list of things to consider:

• Virtual threads and pinning
• Concurrency behavior
• Isolation behavior, including service loading delegation
• JPMS module support (to what degree? can we future-proof?)
• Memory-mapped archives - feasible or not? Is this a viable concurrency solution?
• In-IDE behavior
• Class loading for various kinds of launch modes
• Native image class loading
• Security (JAR signing?)
• Class loader lifecycle/closure
• Relationship to Maven and Gradle
• Relationship to dependency artifact resolution (tighter integration?)
• Relationship to test class loading (see Working group proposal: Test classloading #41867, also consider: split package support?)
• Layering behavior: build time, run time deps, run time app
• Development mode and partial reloading of CL environment
• Any API promises that our current class loaders are making that need to be upheld or deprecated
• Lessons learned (positive and negative) from the WildFly implementation

The third phase would involve prototyping some kind of centralized class loading system which would be adaptable to all of the applicable scenarios above.

The final phase would be integration of the new class loading system and reorganization of the existing code around it.

Expected outcome

The expectation would be that a variety of problems which relate to class loading would be solved, and that future improvements (particularly those relating to JDK restrictions around FFM integration, reflection, and serialization, as well as JDK improvements such as the Project Leyden efforts) would be more possible and/or simpler compared to today.

There is also an expectation that class loading performance would be on par with today, if not better.

As a knock-on benefit, whether or not JPMS support is included, we should be able to provide users with more useful stack traces which include artifact IDs and versions for dependencies.

Related issues (non-exhaustive)

• Classloader optimisations for resource loading #13256
• Class loader cleanups (phase 1) #43726
• RunnerClassLoader - Some classes are loaded multiple times #42874
• Some classes loaders are not ARM compatible #43415
• Mitigate concurrent classes definition in RunnerClassLoader #43022
• Load Testcontainers parent first and start the containers with the Platform CL as the TCCL #42560
• Fix ClassLoader leak in KeycloakDevServicesProcessor #42338
• Replace read/write lock in JarResource to avoid virtual threads pinning #40942
• Netty finalizers load classes from closed class loaders #41607
• Class Loader still used while it's supposed to be closed #41233
• More classloaders cleanup #40557
• Remove deprecated classloader helpers #40564
• Improve loading of classes in RunnerClassLoader #40033
• Most class loader implementations aren't parallel-capable #39952
• Discontinue support for *JarRequiredBuildItem #44518
• Make ResourceUtils public smallrye/smallrye-common#372
• Adding classpath entries to built Quarkus application during startup #44579

[cescoffier]

How does it differ from the working group https://github.com/orgs/quarkusio/projects/30? Should it be merged?

CC @holly-cummins

[holly-cummins]

I think it's reasonable to keep these separate, in the interests of having smaller scopes. https://github.com/orgs/quarkusio/projects/30 covers test classloading specifically, whereas the proposed WG has the broader remit of "all classloading."

However, since there's a substantial overlap, I'd put in a plea to sequence the working groups, and not start serious work on the broader classloader rewrite until https://github.com/orgs/quarkusio/projects/30 is mostly done. Otherwise there's likely to be a lot of rework-cost.

[dmlloyd]

I agree, this is a separate but overlapping effort. And +1 to getting the test stuff done sooner. Having the test class loading rework complete first should simplify this effort, at least to a degree.

[dmlloyd]

Dynamic module loading with JPMS support does seem possible; here's a fairly minimal prototype modeled after the jboss-modules philosophy which implements it and can launch itself: https://github.com/dmlloyd/modules2

[jbonofre]

I'm investigating to support "dynamic loading" with classloaders management.
I started Quaraf (similar to Apache Karaf services but powered by Quarkus).

The idea is to focus on Karaf key features (hot deployment/deployers, classloaders management, service registry/discovery, ...) by leveraging Quarkus.

So, happy to participate in this discussion. I will share on https://github.com/jbonofre/quaraf soon.

[triceo]

Perhaps off-topic here, but was caused by a recent classloading change:
#44283

[franz1981]

I was looking at https://github.com/quarkusio/quarkus/blob/5a2cb66525dc197b2c71ba8bf72eef4e298ef370/independent-projects/bootstrap/runner/src/main/java/io/quarkus/bootstrap/runner/SerializedApplication.java and:

1. I would like to avoid the big amount of single-threaded work performed there to produce the maps/sets etc etc which are used by the runner class loader
2. I would like to test if mmap is beneficial, without assuming it

The reason for 2 is because many database (e.g. levelDB) has moved away from it for good reasons:

1. mmap would like pages to "stay still" and not causing to be read from disk multiple times. And you would like sharing these read pages (across multiple processes). Sadly if you have a container and you won't put the classes in a shared layer, this is not going to happen, and each container instance will read the same data again and again, defeating the purpose of mmap i.e. caching.
2. since OS page cache in the kernel is outside of the application control, the LRU policy could be not as good as we want and "noisy neighbours" can cause OS pages re class loading to be "evicted" and we risk to read them N times from disk even during the lifecycle of the same process. This "transparent" behavior is a known cause of so-called long time to safepoint - which usually break badly the JVMs (see https://groups.google.com/g/mechanical-sympathy/c/LFrJPhyVOJ4)
3. given 1 and 2 if we optimize for NOT sharing, having DIRECT_IO will give much more control on RSS/memory usage and would save copying from disk -> OS pages just performing a single copy when needed (likely over the same tmp off-heap buffer), just to define/link classes. This would allow too to release such buffer when not needed, enabling RSS to be lowered (we need malloc autotrimming enabled for such, since it is not what glibc malloc does by default while you free vs it AFAIK)
4. munmap does something nasty with locks which slow down process exit (not nice, for the JVM, mostly) - see https://elixir.bootlin.com/linux/v6.11.8/source/mm/mmap.c#L3401-L3452 if I got it right

In short; we should think about the use case (now and in the future) where we expect it to be beneficial, before designing things around its strenghts and getting bitten (when it's too late) by its pain points.

[geoand]

Correct, just the dat file

[dmlloyd]

My thought, at least for one launch mode, is to have one large JAR with the whole application in it, possibly containing other ("stored", not compressed) JARs whose classes are also not compressed, and mmap the single JAR at once. The resource loader would then return sliced buffers when classes and resources are loaded. Finally we'd madvise it away after startup completes.

But, since we would hide it all behind the resource loader API, the class loader is not tied to one strategy, which will let us compare many approaches.

[Sanne]

I would like to avoid the big amount of single-threaded work performed there to produce the maps/sets etc etc which are used by the runner class loader

Right, we discussed this 6+ months ago :) Should be straightforward, just need someone with some spare time™ to want to have some fun, and it would be clearly beneficial on its own (irrespective of the rest of the discussions here).

if you have a container and you won't put the classes in a shared layer, this is not going to happen, and each container instance will read the same data again and again, defeating the purpose of mmap i.e. caching.

As you well know that's totally my intention to make sure that at least a good chunk will be stored in a common layer, shared across applications - including different applications but with the assumption that they run the same Quarkus version. This might not necessarily be the only way of running a Quarkus application but it should be the strategy we optimise for.
But we'll never be sharing the entirety of the classpath, so I'd like to see this approach applied tactically on the core layer, but not (necessarily) on all classloaders.

.. and we risk to read them N times from disk

TBH I don't see this as a major issue. Saving memory and sharing memory should be the priority even over slightly slower bootstrap; and being in a position to potentially share memory in several cases is better than not sharing such memory just because it might not always be effective.

.. having DIRECT_IO

Sure, that's interesting and perhaps better.

[Sanne]

My thought, at least for one launch mode, is to have one large JAR with the whole application in it, possibly containing other ("stored", not compressed) JARs whose classes are also not compressed, and mmap the single JAR at once. The resource loader would then return sliced buffers when classes and resources are loaded. Finally we'd madvise it away after startup completes.

@dmlloyd +1 But could I have two "jar"s ? I'd like to have a single blob for the bootstrap classloader, and one for the rest of the application (user code + extensions). The idea being that multiple, different Quarkus applications would be able to share the boostrap layer; this layer would potentially be combined with the jdk's base modules, and optionally pre-trained for Leyden, etc..

[dmlloyd]

My thought, at least for one launch mode, is to have one large JAR with the whole application in it, possibly containing other ("stored", not compressed) JARs whose classes are also not compressed, and mmap the single JAR at once. The resource loader would then return sliced buffers when classes and resources are loaded. Finally we'd madvise it away after startup completes.

@dmlloyd +1 But could I have two "jar"s ? I'd like to have a single blob for the bootstrap classloader, and one for the rest of the application (user code + extensions).

I think we'll be forced to split it up to an extent, because the launcher itself must be a regular module on the module path I think. We can evaluate different things to see how well they work.

But I also think that we should consider multiple use cases (not just containers) and make sure we have appropriate launch modes for each use case. Shared layers can be antithetical to whole-program optimizations if we're not careful in how we approach it. Cases where we want to maximally share might see a greater benefit from sharing than what is lost in optimization. But when we do not share, the opposite may be true.

But in any event, the first step is the first step, and that is: can we fundamentally change how we launch Quarkus to accommodate more flexible and powerful layouts? So I think the first thing to attempt will be to create a launch mode which just launches a Quarkus app using the prototype modular indexed class loader, and see what can be learned from that. If this is proven to be viable then we should be able to explore a wide variety of layout options.

[franz1981]

@dmlloyd Just for fun I've tried this -> https://github.com/franz1981/quarkus/tree/shared_dir_names
Because I've noticed that both hashCode and substring duplications
eg

dir org/eclipse/microprofile/config ascii? true
subdir org
subdir org/eclipse
subdir org/eclipse/microprofile

have some common forms (as expected):

1. mostly ascii - I used this information available at build time to save some String check and slow readUTF calls (which perform 3 copies; byte[] -> char[] -> String's byte[] vs just byte[] -> String's byte[])
2. the subdirs can be obtained from the full dir name
3. there's no point to compute the hashCode of the de-serialized sub-dir names since they can be computed and saved while producing it - in Java the hashCode number is part of the spec!

For example, in a normal non-crud quarkus APP we have 962 full dir names and 2959 sub-dirs from them - which is ~3 * full dirs.
Sadly java doesn't allow to lookup maps with a custom hash code / equals per-map, and that forced me to allocate a cheap StringView (which is ~16 bytes) per each lookup :"( or I would have used the original String - computing the substring hashCode on the flight by sanitizing the "bad" chars (.) - saving both to create the cheap StringView instance AND the replaced String one - as we do now.
I'm curious to verify what's the improved footprint - it was really a run for low hanging fruits there, so maybe it's not even worthy.

It seems, doing some basic math, that I'm getting a saving of 30-40% with this vs original behaviour, in the old generation (since these maps usually fall there) - said that, it's not bad.

[franz1981]

And it seems I've missed a thing: the ResourceDirectoryTracker::addResourceDir doesn't make ALL the subdirs to be added to the resource map - but a tiny fraction (in my case it was 4K dir/subdirs read vs 1K actually stored.
It's likely that we can still "do it right" here, by saving useless StringView allocations, by making it mutable.
In this way reading the StringView out of the serialized form won't allocate anything unless it is retained in the resource map.

[dmlloyd]

Resource loading maps for JAR files are very interesting, but also quite tricky.

Every JAR file path is stored as a whole path. Meaning, the same strings are repeated hundreds or thousands of times in the JAR file image, and, if we're not careful, in memory as well. Furthermore, directories sometimes have entries and sometimes do not, but we nearly always need a logical concept of a directory for each resource no matter what.

As you say, we can make things more efficient by not copying the strings out of the JAR index, and this works in a lot of cases. I believe that java.util.jar.JarFile is doing this in fact - storing hash codes and offsets of the entries, but not copying the strings themselves. This works great as long as you only care about file lookup and not directory iteration.

For directory iteration, one choice would be to iterate all entries in the JAR for every directory, filtering out all entries that represent files that are not directly present in the directory we're iterating. To support subdirectory iteration, you would also need to track each file in a nested directory that we encounter in case the corresponding directory doesn't have an explicit zip entry. This starts to get costly. We're doing most of this in SmallRye Common now. But we're not using those APIs in Quarkus much yet.

One strategy I've been thinking about is a highly compact tree structure. Each level of the tree would represent a directory, and would contain sorted tree entries for each subdirectory and file in that directory. In memory, the tree would be a single flat integer or byte array whose values contain pointers to the file name, zip entry (if any), and/or nested subdirectory. A binary search would be used to locate entries.

While this looks good in theory, it's pretty tricky to make it actually be good. In many cases, though the file names might be 40 or more characters long, the amount of space saved might be a lot less for a given file when you account for all of the nested tree levels. A binary search like this also would entail more pointer chasing for location of entries by whole path. On the other hand, directory iteration becomes very efficient.

One of the nice things about relying on the SmallRye Common resource loading API in Quarkus though is that we would be able to try out lots of implementation ideas right away before we commit to anything.

[dmlloyd]

We can reasonably split phases 2-4 on parallel tracks:

• Class loading within Quarkus itself
• Class loading within tests (covered by another WG right now)
• Class loading within generated images

Tackling the generated images first seems like an achievable goal. There are definitely things to discuss around artifact resolution (and categorization) that directly pertain to class loading and isolation, but we should be able to produce at least a POC of a modular generated image to see what blows up, if we're not afraid to hard-code a few things.

[cescoffier]

I like the idea of splitting the problem into smaller (but still challenging) ones. As I said, I prefer a sequence of short working groups to a large one where "progress" is hard to measure.

[dmlloyd]

Tasks for the project board (if and when; updating as needed):

• More classloaders cleanup #40557
• Remove deprecated classloader helpers #40564
• Static assets overwrite #41449
• Class loader cleanups (phase 1) #43726
• Discontinue support for *JarRequiredBuildItem #44518
• Move recorded bytecode into a runner package #44655
• Generated classes are not associated with an artifact #44657
• ResolvedArtifactDependency contains no dependencies #44659
• Clean up split packages between deployment and run time artifacts #44710
• Make ResourceUtils public smallrye/smallrye-common#372

[dmlloyd]

The WG is still pending, but here's a summary of the research so far.

My prototype branches are here:

• https://github.com/dmlloyd/quarkus/tree/modular
• https://github.com/dmlloyd/modules3

The theory of operation is:

• Each dependency/artifact gets its own class loader and module
  • The class loader/module is created lazily as classes are loaded
• Artifacts which have a module-info are created as named modules
  • If artifacts are also marked as "parent-first", I put it on the boot module path which is loaded by the JDK (this is kind of a weird hack but it works)
• Other artifacts are created as "unnamed" modules, even if they have an automatic-module name
• There are a few other details around service loading, resources, and how modules are created by the JDK

What I've learned so far:

• It works, basically, though there are a few other things to do
  • I still need to perform about 6 or 7 manual module patches after a build to get an app to start up
• Split packages are a problem (though not as bad as it may seem by looking at Clean up split packages between deployment and run time artifacts #44710)
• Generated classes are a problem (Generated classes are not associated with an artifact #44657, this is more serious)
• I've created MODULAR as a JAR packaging type, but it might be better to have an extension instead which disables JAR generation and generates module output in addition to JAR output. Not sure yet.

Next steps:

I think that creating the modules on demand is a good design philosophy (inherited from jboss-modules), however it comes at a cost, specifically that service loading doesn't always work exactly correctly according to the JPMS. I think the next stage of prototyping will involve creating a single ModuleLayer for all of the named modules, instead of a separate ModuleLayer for each one. This will allow service loading to work a lot more predictably. It might also allow us to treat automatic modules as named modules (though I'm not certain on that yet).

Mixing named and unnamed modules actually works pretty well. This is good because a number of artifacts do not yet define module-info.class. In the getting-started example, we have 72 out of 111 JARs which are not modular (though that includes JARs that would be automatic modules).

I'm going to move to another task for next week and come back to this a bit later.

[Sanne]

This all sounds great, but we also need to balance the tradeoffs with the fact that Leyden won't be of use on any custom classloader - at least for the time being.

I'm +1 for this plan as I don't expect us to be able to enable much benefits from Leyden on all extensions and 3rd party libraries, but also I hope we could load all core and bootstrap components within the system classloader, so to benefit from Leyden in a shared layer which includes the JDK modules and indeed the code which will implement all the module classloaders by @dmlloyd .

Essentially I hope we'll be able to use two different strategies, combine with David proposes here for all extensions and user code, with a Leyden non-modular core and this should maximize benefits - although I don't have evidence for this yet I suspect we agree that this is most likely going to proof an effective combination?

[dmlloyd]

I am more or less ignoring Leyden (which AFAICT means "preloading stuff into CDS" these days) for the purpose of this work, but you are right that the core layer will be loaded by the system class loader and thus this can theoretically be utilized for Leyden (this layer, however, is 100% modular; I'm not sure what you mean by "non-modular core"). This comprises a relatively small percentage of the overall code being included at present though, which is something that can be looked at separately (it is good or bad depending on your point of view).

Since it's still early stages I can't say for sure how well this will or will not work with Leyden as it stands today. We'll have to try some things out and see what happens.

[Sanne]

Perfect, sounds like we're on the same page 👍