User initialization task support

[iocanel]

Resolves: #32123
Resolves: #34643

The pull request introduces an annotation @PreStart that is meant to annotate Runnable classes to mark that they should run during the initialization phase of the application (before Startup).

Updates:

• Flyway, Liquibase and Liquibase Mongo adopted @Initialization
• Each initialization task is masked to a different Kubernetes Job.
• Add support of task filtering.
• Allow non-Runnalbe tasks (the runnable is generated).
• Rename @Initialization to @PreStart.
• Added integration test that ensure that user provided task are executed

[iocanel]

cc @kdubb

[iocanel]

The pull request does work, but it's currently affected by #32729

[geoand]

Is this meant to cause the exit of the application when the task is done?

[iocanel]

No

[iocanel]

This just tells quarkus, that the annotated code needs to be executed during initalization.

[geoand]

How is this intended to be different than https://quarkus.io/guides/lifecycle#startup_annotation?

[iocanel]

The main differences are the following:

• Init tasks can be globally enabled / disabled.
• Init tasks run before the StartupEvent is fired (it's guaranteed that they all run before the app).
• Extension services may depend on the completion of an Initialization tasks (before the register the ServiceStartBuildItem).
• Init tasks are mapped to Kubernetes Jobs.

[geoand]

Init tasks can be globally enabled / disabled.

What's the use case for doing this?

Init tasks run before the StartupEvent is fired (it's guaranteed that they all run before the app).

Same question as above :)

Extension services may depend on the completion of an Initialization tasks (before the register the ServiceStartBuildItem).

When would this happen in practice?

[iocanel]

An other way to put is that that Initialization is meant to be used for code that can be optionally run isolated (e.g. as kubernetes jobs or via the cli) to prepare the ground for the actual application. Startup seems more like a listener to me.

[geoand]

Okay, that's an important distinction.

But the name of the annotation does not convey this at all, it's far too generic IMHO

[iocanel]

I think that at Initialization is a pretty good name:

• It rhymes with at Startup
• Initialization also implies before startup (this is the main distinction with Starutp IMHO).

Everything else has to do with how we use it, so I don't feel that the name should convey these.

[geoand]

My point is that it's far too generic - it does not in anyway convey that this piece of code could be "standalone"

[kdubb]

It might be good to record some motivation...

@iocanel and I discussed this privately and the feature comes from a pattern we noticed internally in our team. We needed to prepare our DevServices (think integration testing) and we did that with some custom code and QuarkusTestLifecycleManagers. Then we would essentially recreate that in scripts to prepare the same services for production in K8s. It quickly became obvious this was the same code and should be shared instead of being written twice in two different languages. The other realization is that nearly every external service (and therefore DevService) required some amount of preparation.

We solved this manually internally by creating a Quarkus CLI application (to be used as a K8s init Job) and rearranging our code to run our tasks at startup (using @startup) or manually via the CLI.

This is essentially what Flyway & Liquibase do for databases. When @iocanel implemented the K8s init job feature for Flyway it sparked the discussion for this feature as a similar but more generic service initialization/preparation framework.

This PR has serious advantages, the obvious being you don't have to do this manually anymore. Additionally, that the tasks are run before the application is started instead of after @startup fires which can cause timing/expectation issues.

[kdubb]

@iocanel Thanks for working on this!

[geoand]

Thanks for providing the context @kdubb.

This PR has serious advantages, the obvious being you don't have to do this manually anymore. Additionally, that the tasks are run before the application is started instead of after https://github.com/startup fires which can cause timing/expectation issues.

I have no doubt of this. I just want us to have proper naming and documention because I am 100% sure that users will easily be confused on what each piece does when generic names are used

[kdubb]

Maybe @ServiceInitialization?

[geoand]

That is definitely better :)

[iocanel]

I honestly don't feel that Service as a prefix adds anything here. In addition it's less comparable to Startup.
If the name needs to tell something to users is that it runs before startup. 😄

[geoand]

In addition it's less comparable to Startup

That is feature in my mind. I am thinking of our users here: If they come across @Startup and @Initialization for the first time, then they will absolutely be confused

[iocanel]

I am thinking of our users here: If they come across @Startup and @Initialization for the first time, then they will absolutely be confused.

Agree, it's easy to get confused, as their purpose is very similar. The part where I disagree is that naming will demistify things.

[geoand]

If we agree it's easy to get confused, why choose a generic name?

[iocanel]

If we agree it's easy to get confused, why choose a generic name ?

I don't find it generic.
How would you name an annotation that expresses that the annotated code should run during the initialization phase?

• ServiceInitialization? Service does not mean anything in this context.
• ApplicationInitialization? Better than ServiceInitialization, yet the Application prefix doesn't add anything. It's also inconsistent with @Startup that doesn't bear the same prefix.
• BeforeStartup ? Ugly, but demisitifies ordering of things and is consistent with Startup. It could work.

Still prefer Initialization but I could see BeforeStartup working.
Other suggestions?

[github-actions]

🎊 PR Preview ca75503 has been successfully built and deployed to https://quarkus-pr-main-32728-preview.surge.sh/version/main/guides/

[geoand]

How would you name an annotation that expresses that the annotated code should run during the initialization phase

The way I see it, the term initialization phase is pretty overloaded itself.

If we are going for something like @BeforeStartup, maybe @PreStartup is a tad better?

[iocanel]

The way I see it, the term initialization phase is pretty overloaded itself.

Not sure what you mean by overloaded. From my pov its the most common term to describe settings things up before the application starts up.

Besides that, our code is already using it, our config properties are already using it (see quarkus.init-and-exit). Kubernetes is already using (see initContainers). I just can't see why we would name the annotation differently. IMHO, it's not a matter of tase. It's wrong to use anything else.

[geoand]

It's wrong to use anything else

Let's just agree that we totally disagree :)

Kubernetes is already using (see initContainers)

Sure, but I am looking at it from a different perspective - I am looking at it from the perspective of the maintainer that answers countless user questions and as a result has become fairly competent at predicting user experience issues.

[geoand]

I should two things that might not be clear from the conversation above:

• I think this feature is super useful
• I would not be complaining about the name if we didn't already have the @Startup annotation

[iocanel]

@manovotn: Thanks for taking the time to share your thoughts.
You touched on many areas but it feels like the main theme is why not use: Startup / StartupEvent, so I'll focus on that for now.

The main reasons are:

• I can't guarantee that init tasks will be executed before the application code.
• I can't guarantee that I can optionally exit when all initialization tasks are completed.

Using priorities two enforce the above, is not trivial and it feels like something users will use to shoot their foot.
As initialization tasks come from multiple sources (e.g. extensions, user code etc) orchestrating them also sounds non-trivial and will require users to implment on their own something similar to what this PR suggests.

[manovotn]

Using priorities two enforce the above, is not trivial and it feels like something users will use to shoot their foot.

It uses the same logic that interceptors have been using for years - there are predefined priorities that determine, roughly, when your interceptor should run. This is used for CDI observers too and could be used here as well.
See https://github.com/jakartaee/interceptors/blob/master/api/src/main/java/jakarta/interceptor/Interceptor.java#L102-L129

On top of that, assuming this goes into the code, the next ask may be for ordering of these kinds of init tasks (which is BTW exactly how we got to ordering observers in CDI) and suddenly you have the exact same "shoot in the foot" problem/argument.

I can't guarantee that init tasks will be executed before the application code.

Not sure I follow here. Your init tasks can even exit when all tasks are completed yet are allowed to run after application code?
In any case, the ordering I mentioned can still be used here - using priority to execute them later. Alternatively, you can even have an observer for a custom event type that your app fires whenever this should be executed, that's also an option you already have.

I can't guarantee that I can optionally exit when all initialization tasks are completed.

Could you clarify in what scenario you need this? I did read the comments but I am still failing to grasp it.
Especially in what case do I need to run task and exit app versus having an init task that's setup to "run once and skip on every following startup".

[kdubb]

I provided insight into what our usage of this would be quite a while back and I'm just catching up on the discussion but I think our use case is very valid and would be of use to anybody building applications that go beyond "getting started".

Let me take you through a specific example that plagues us today...

DevServices provide a nice DX but there is almost no way to initialize them with the required configuration beyond what is provided in the devservices configuration. We use RabbitMQ to the point I contributed the RabbitMQ connector for Quarkus. I knew we needed a complex set of pre-defined exchanges/queues in our application, so I added a lot of this to the capability of RabbitMQ's devservices implementation by way of configuration.

Unfortunately, we needed more configuration capability because the SmallRye Reactive Messaging configuration is pretty limited. Rather than attempt to contribute more configuration capability to the RabbitMQ extension (which would be an ever growing list of configuration knobs) we took the route seemingly suggested by @manovotn and others and wrote code that runs during @Observes Startup.

This works but has two major pain points:

1. We cannot do it until after all the extensions are "running".
2. We cannot do it until after the application starts.

The first problem, running initialization after all the extension are already "running" is a nuisance (and a pretty bad one). For the case of "SmallRye Reactive Messaging RabbitMQ", this means that our "unconfigured" dev service instance gets started, then SRRM immediately starts connecting to the RabbitMQ "unconfigured" instance and starts endlessly logging connection errors due to the (required) retry configuration. Finally, we receive the @Observes Startup event which runs our initialization task for RabbitMQ and then SRRM can connect properly.

The second problem, running after the application has started, becomes kind of obvious. The application is in a "started" state but it has not completed initialization. It seems to me that the initialization of devservices should be completed before the application is considered to be started. Not doing this causes issues with tests specifically.

Essentially we want a "Flyway" for RabbitMQ...

A configuration step that is linked to the extension and can contribute to initialization before anybody considers the application "started". If it's a valid case for database initialization then it's a valid case for message brokers.

Now repeat the above for Vault...

We have the exact same issues with Vault. The Vault extension could never provide enough configuration points for us to configure it properly. We need to ensure RabbitMQ and PostgreSQL are initialized with credentials and then store those in Vault. Clearly this is not something that devservices could provide "configuration" for.

What do we do with all this initialization code???

Prior to us building our initialization tasks, we used (as much as we could) the capabilities provided by the devservices implementations, then we added code to @BeforeAll and began using QuarkusTestResourceLifecycleManager to ensure our devservices containers were "properly" initialized. This worked fine.

Then we had to go and recreate all this configuration for a production app that we ran in two places, via docker compose for verification testing and ultimately in Kubernetes. Which means we had two methods of setting up our services externally and all the stuff I just described scattered through our configuration and test code.

Initialization tasks nicely cleaned all this up. The "aha" moment was when we realized that this same code could be broken out, without any changes, into a Quarkus command line app and run as an initContainer in Kubernetes.

Initialization tasks allow us to manage, in code, the initialization of our external services; wether they be "devservices" in Docker or "real services" in Kubernetes.

[kdubb]

Just to add to my already longwinded response...

I hope you can see that this becomes a pattern. Each external service needs some initialization for testing and production. Adding a formalized way to achieve this in Quarkus will be a stand out feature that provides an exceptional developer experience that works for development, testing and production.

[kdubb]

One thing I forgot to touch upon was ordering; which is a valid concern @manovotn brings up.

We currently have initialization tasks for Vault, RabbitMQ, OpenFGA and Redis. Currently one of the tasks is cross-cutting, Vault.

For 'dev' and 'test' the Vault task needs to connect directly to PostgreSQL and RabbitMQ to configure them both for Vault's dynamic credentials. So the task needs to be sure that those containers are running. Currently when running during @Observes Startup or in K8s we assume this to be true; because it is.

When formalized as a feature I think must still be true, otherwise you are limiting an initialization task to only working with a specific service.

This could be fairly easily remedied by having specific sequence points.

1. DevServices container startup
2. Initialization tasks execution
3. Extension startup
4. Application startup

That would allow any initialization task to configure inter-working services as necessary.

[iocanel]

I can't guarantee that I can optionally exit when all initialization tasks are completed.

Could you clarify in what scenario you need this? I did read the comments but I am still failing to grasp it. Especially in what case do I need to run task and exit app versus having an init task that's setup to "run once and skip on every following startup".

Tasks like flyway, liquibase etc may run isolated from the app (e.g. run once as Kubernetes Jobs). This is something we already do and this PR intends to make this feature available to user init code too.

[mkouba]

What is the status of this pull request?

I'm not so sure this API would be very useful. I mean most of the users would not need to execute some logic before the app actually starts and the extensions should make use of the build items and recorders.

If I understand it correctly the use case is to guarantee that some logic is executed before the StartupEvent is fired and recorder methods invoked from build steps consuming the ApplicationStartBuildItem are executed. If that's the case then @Observes @Priority(-1) StartupEvent event should do the trick because most of the extension observers should have the priority value between jakarta.interceptor.Interceptor.Priority.PLATFORM_BEFORE and jakarta.interceptor.Interceptor.Priority.LIBRARY_BEFORE. @kdubb For example, if you need to execute some logic before the SR Messaging is started then just use @Observes @Priority(aValueLessThan1000) StartupEvent because of https://github.com/quarkusio/quarkus/blob/main/extensions/smallrye-reactive-messaging/runtime/src/main/java/io/quarkus/smallrye/reactivemessaging/runtime/SmallRyeReactiveMessagingLifecycle.java#L43-L64.

In fact, the current logic in InitializationTaskProcessor.startApplicationInitializer() does not seem to guarantee the requirement mentioned above at all. As it only consumes the SyntheticBeansRuntimeInitBuildItem and InitTaskCompletedBuildItems which implies that is must be executed after synthetic beans are initialized and all build steps producing InitTaskCompletedBuildItem are executed, i.e. no requirement that it has to be executed before app start (but it probably works by coincidence).

[kdubb]

@mkouba How is the @Observes then packaged into a K8s init container?

[mkouba]

@mkouba How is the @Observes then packaged into a K8s init container?

@kdubb I don't know what you mean with "packaged into a K8s init container" but I guess that it would be packaged in the same way as the "init task" introduced in this PR?

[kdubb]

@mkouba That was my point. This PR is essentially working to make all this a "supported feature" via init tasks. Your comment focused on one single aspect of that "running before main" but when viewed as a whole this PR I think is where we end up.

[mkouba]

@mkouba That was my point. This PR is essentially working to make all this a "supported feature" via init tasks. Your comment focused on one single aspect of that "running before main" but when viewed as a whole this PR I think is where we end up.

Well, as I mentioned in the comment the @Observes @Priority(-1) StartupEvent (or @Startup(-1) void init()) should work in most cases. So it's not focused on single use case.

Speaking of this PR - I don't think it's guaranteed that the init tasks will be executed before the StartupEvent. The relevant build steps would need to produce a ServiceStartBuildItem.

Also I don't understand why the @PreStart annotation is a qualifier? It does not seem to be used as a CDI qualifier if declared on a method.

[kdubb]

So it's not focused on single use case.

This conversation is just going around in circles. At this point I'll just exit saying this... Quarkus's tagline starts "A Kubernetes Native Java stack...". App initialization is a huge problem in need of solving (I have detailed many examples already); and indeed Flyway was deemed special enough to get this treatment. This feature aimed to solve that generally and provide first class support for initialization of any service that could be shared across development, testing and (K8s) production. Wether it's this PR or another this feature should land.

We are currently rolling our own shared initialization (again as described in detail in previous posts), so I know firsthand how hard it is to figure out; including all the warts that currently exist. Quarkus should help here given it's goals.

[geoand]

This conversation is just going around in circles

From the little I have observed, I agree. However I have not had the time to dig it, but I am hoping that sometime next month I will.

[mkouba]

This conversation is just going around in circles. At this point I'll just exit saying this...
We are currently rolling our own shared initialization (again as described in detail in previous posts), so I know firsthand how hard it is to figure out; including all the warts that currently exist. Quarkus should help here given it's goals.

I'm sorry I'm very likely missing some important piece of information. However, I don't disagree that Quarkus should help in this area...

It would be great if we could (1) summarize all the requirements, (2) describe the current solution (which IMO currently does not guarantee what it declares) and (3) try to propose a new API based on 1. and 2.

[mkouba]

Current solution

There is a need to perform some environment initialization logic (env init) in a quarkus app. Typically, in order to be able to execute a quarkus app as a Kubernetes Job. In this particular case, we also need a way to exit the app once the env init completes but before the application starts (init-and-exit).

An extension can produce an InitTaskBuildItem that represents an init task and is consumed by the Kubernetes extension. An extension can produce an InitTaskCompletedBuildItem in a build step that also calls a recorder. It should be guaranteed that the recorder is called before init-and-exit is performed and before the appllication starts (StartupEvent is fired). A user cannot leverage this API.

Requirements

We need a new API to define env init logic in CDI beans.

• It should be possible to mark a bean as an "env init task" and assign a unique task name/id
• It should be possible to skip the execution of the env init logic completely (this PR makes use of the quarkus.init-disabled config property)
• It should be possible to skip execution of a specific env init task (this PR makes use of the quarkus.init-task-filter config property)
• The logic defined in the bean should be executed before init-and-exit and before the appllication starts

New API

We could introduce a new annotation or interface to mark a bean as an "env init task".
An interface implementation might require less magic:

public interface InitTask {
   String getName();
   void run();
}

User code:

@ApplicationScoped
public class MyBean implements InitTask {
    public String getName() { return "foo"; }
    public void run() {}
}

Internal usage:

@Inject
Instance<InitTask> tasks;

public void runAll() {
    tasks.forEach(InitTask::run);
}

[kdubb]

@mkouba Please read my previous comment (#32728 (comment)), it's a fairly detailed use case

in order to be able to execute a quarkus app as a Kubernetes Job.

This isn't really the requirement. In order to run a Quarkus application as a Kubernetes Deployment/Statefulset the environment needs initialization. The initialization tasks you propose (i.e. your InitTask) need to be run during dev/test and then for production, in K8s, they need to be gathered and ran in a separate Quarkus CLI app that exhibits the init-and-exit behavior.

[kdubb]

Also, as outlined in my previous comments, these InitTasks need to be run after all DevService containers have started but before the application or extensions "start". This requirement ensures dev/test see a similar view of the environment as when running in production.

[mkouba]

@mkouba Please read my previous comment (#32728 (comment)), it's a fairly detailed use case

in order to be able to execute a quarkus app as a Kubernetes Job.

This isn't really the requirement.

That's why I used "Typically".

Also, as outlined in my previous comments, these InitTasks need to be run after all DevService containers have started...

@iocanel Is this requirement implemented in this PR?

but before the application or extensions "start". This requirement ensures dev/test see a similar view of the environment as when running in production.

Hm, I'm not sure there is a way to formalize the extensions "start" part. What exactly do you mean?

[kdubb]

As an example, on our pain points is with SmallRye Reactive Messaging. We need to setup the RabbitMQ instance with dynamic credentials before it will work and this requires both the Vault and RabbitMQ containers to be running. Unfortunately SRRM begins attempting its connections before application start which creates a lot of logged errors during startup in dev/test.

This means to properly initialize the Vault and RabbitMQ containers, without spurious errors, the InitTasks would need access to these containers before the application or extensions begin to use them.

Generally speaking this means the InitTasks should have access to all the dev service containers prior to any code that attempts to use them.

[kdubb]

We also have the same exact problem with Postgres and dynamic credentials.

[mkouba]

Generally speaking this means the InitTasks should have access to all the dev service containers prior to any code that attempts to use them.

From the implementation point of view we could probably just consume the DevServicesLauncherConfigResultBuildItem in the env init task build steps.

[kdubb]

That seems correct. Wouldn't others need to then consume some new InitTasksCompletedBuildItem to ensure they are running post InitTask execution?

[mkouba]

That seems correct. Wouldn't others need to then consume some new InitTasksCompletedBuildItem to ensure they are running post InitTask execution?

Well, the env init task build steps should also produce ServiceStartBuildItem so that it's guaranteed that init-and-exit is performed before the application start. What "others" do you mean?

[kdubb]

I'm no expert here. I've contributed quite a bit to Quarkus but that list of build items is... big 😏. ServiceStartBuildItem seems like is already providing this.

Back to the SRRM example, I'll look into if that is consuming ServiceStartBuildItem or DevServicesLauncherConfigResultBuildItem.

[kdubb]

After a quick glance at the SRRM extension I can't find anything that would seem to order it after even dev services. Maybe that explains why it's initializing so early in the startup sequence.

Is ServiceStartBuildItem something that all these service connecting extensions should be consuming?

[rteabeault]

Found this PR when looking for this type of functionality and it appears to have died on the vine. In my use case I can not use the existing Liquibase extension because our datasources are not static at build time but dynamically read at runtime. However, we wanted to make use of the existing QUARKUS_INIT_AND_EXIT to run Liquibase as an K8s job. So my choices without this PR seem to be

1. Create my own Liquibase extension so that I can run during init. Or
2. Use @Startupand define my own type of config like QUARKUS_INIT_AND_EXIT that allows me to exit after.

My own extension seems like the right thing to do currently. However, with this PR I could create my own Liquibase runner that runs at init time and make use of QUARKUS_INIT_AND_EXIT.