Quarkus Quartz: Quarkus 1.13.x scheduler is not updated in DB when updated after upgrading to Quarkus 2.x #22479

ruiarodrigues · 2021-12-22T16:14:22Z

Describe the bug

Hi everyone!
Some months ago, some schedulers were created using Quarkus 1.13.x and were deployed to production. Everything worked fine. Some weeks ago, we had the need to update the cron job of one of the scheduler but found out that the scheduler was updated in local tests but not in production. Currently we're running Quarkus 2.5.1.

After some digging in Quartz scheduler code, we found out that Quartz is not able to find the trigger because trigger name has changed from Quarkus 1.13.x to 2.x.

The query performed by Quartz is (from org.quartz.impl.jdbcjobstore.StdJDBCDelegate#selectTrigger)
"SELECT * FROM QUARTZ_TRIGGERS WHERE SCHED_NAME = 'QuarkusQuartzScheduler' AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ?"

Below one can see that the column trigger_name from QUARTZ_TRIGGERS table has different values when the schedulers are created in Quarkus 1.13.x and 2.5.1: _trigger was added to the identity of the scheduler

Quarkus 1.13.7

sched_name	trigger_name	trigger_group	job_name
QuarkusQuartzScheduler	executeDataRemoverJob_trigger	io.quarkus.scheduler.Scheduler	executeDataRemoverJob
QuarkusQuartzScheduler	executeCalculationJob_trigger	io.quarkus.scheduler.Scheduler	executeCalculationJob

Quarkus 2.5.1

sched_name	trigger_name	trigger_group	job_name
QuarkusQuartzScheduler	executeDataRemoverJob	io.quarkus.scheduler.Scheduler	executeDataRemoverJob
QuarkusQuartzScheduler	executeCalculationJob	io.quarkus.scheduler.Scheduler	executeCalculationJob

When doing the query to find the trigger, none is found and no update is done. Since the job_name is the same, no new scheduler is created. Also we deleted one of the scheduler from the code but the trigger is still in the table. I think it's not deleted due to the same problem.

I checked the migration guide to 2.x and I didn't see any mention about this changes in Quartz.
Can you tell me if this is a known issue?

I didn't yet try any workaround but I will try some solution and post it here.

Expected behavior

Schedulers created with Quarkus 1.13.x should be updated when the update is done running the app with Quarkus 2.x.

Actual behavior

Schedulers are not updated

How to Reproduce?

create a Scheduler using Quarkus 1.13.x
upgrade the app to Quarkus 2.5.1
update the scheduler cron in the code
start the app: scheduler still runs using the old cron (DB is not updated)

Output of `uname -a` or `ver`

No response

Output of `java -version`

openjdk 11.0.11 2021-04-20 OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9) OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.5.1.Final

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2021-12-22T16:14:25Z

/cc @machi1990, @mkouba

machi1990 · 2021-12-23T02:48:08Z

Below one can see that the column trigger_name from QUARTZ_TRIGGERS table has different values when the schedulers are created in Quarkus 1.13.x and 2.5.1: _trigger was added to the identity of the scheduler

The removal of the _trigger suffix went in 2.1 release but I do not think we ever documented it, @mkouba do you remember?
The workaround will be to use the identity property on the Scheduled annotation. The value should be the old trigger name.

I think it's not deleted due to the same problem.

Quarkus's Quartz extension does not clean the phantom triggers for you. You could do so with a script using Flyway or Liquibase extensions.

mkouba · 2021-12-23T08:04:52Z

do you remember?

I don't remember. It was probably not documented because we did not think of this use case? identity or an update script is IMO the way to go.

ruiarodrigues · 2021-12-23T08:38:29Z

Thanks for the quick feedback. I'll test the addition of _trigger to the identity to see if it works and I'll let you know.
It would be nice to add this to the migration guide of 2.1 though for those who haven't upgraded yet from 1.x.

ruiarodrigues · 2021-12-23T09:01:48Z

I changed the identity of the scheduler from executeCalculationJob to executeCalculationJob_trigger as suggested but quarkus doesn't start.

2021-12-23 08:54:30.509 ERROR [io.qua.dep.dev.IsolatedDevModeMain] (main)  Failed to start quarkus: java.lang.RuntimeException: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.dev.appstate.ApplicationStateNotification.waitForApplicationStart(ApplicationStateNotification.java:51)
	at io.quarkus.runner.bootstrap.StartupActionImpl.runMainClass(StartupActionImpl.java:122)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.firstStart(IsolatedDevModeMain.java:145)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:450)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:67)
	at io.quarkus.bootstrap.app.CuratedApplication.runInCl(CuratedApplication.java:149)
	at io.quarkus.bootstrap.app.CuratedApplication.runInAugmentClassLoader(CuratedApplication.java:105)
	at io.quarkus.deployment.dev.DevModeMain.start(DevModeMain.java:145)
	at io.quarkus.deployment.dev.DevModeMain.main(DevModeMain.java:63)
Caused by: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:104)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:67)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:41)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:120)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:103)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Unable to create Scheduler
	at io.quarkus.quartz.runtime.QuartzScheduler.<init>(QuartzScheduler.java:221)
	at io.quarkus.quartz.runtime.QuartzScheduler_Subclass.<init>(Unknown Source)
	at io.quarkus.quartz.runtime.QuartzScheduler_Bean.create(Unknown Source)
	at io.quarkus.quartz.runtime.QuartzScheduler_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:96)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:29)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:26)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:26)
	at io.quarkus.quartz.runtime.QuartzScheduler_Bean.get(Unknown Source)
	at io.quarkus.quartz.runtime.QuartzScheduler_Bean.get(Unknown Source)
	at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:440)
	at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:453)
	at io.quarkus.arc.impl.ArcContainerImpl.instanceHandle(ArcContainerImpl.java:424)
	at io.quarkus.arc.impl.ArcContainerImpl.instance(ArcContainerImpl.java:233)
	at io.quarkus.arc.runtime.BeanLookupSupplier.get(BeanLookupSupplier.java:29)
	at io.quarkus.vertx.http.runtime.devmode.DevConsoleRecorder.addInfo(DevConsoleRecorder.java:41)
	at io.quarkus.deployment.steps.DevConsoleProcessor$runtimeTemplates-308161071.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.DevConsoleProcessor$runtimeTemplates-308161071.deploy(Unknown Source)
	... 13 more
Caused by: org.quartz.ObjectAlreadyExistsException: Unable to store Trigger with name: 'executeCalculationJob_trigger' and group: 'io.quarkus.scheduler.Scheduler', because one already exists with this identification.
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1184)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$2.executeVoid(JobStoreSupport.java:1068)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3780)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3778)
	at org.quartz.impl.jdbcjobstore.JobStoreCMT.executeInLock(JobStoreCMT.java:245)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1063)
	at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:855)
	at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:249)
	at io.quarkus.quartz.runtime.QuartzScheduler.<init>(QuartzScheduler.java:200)
	... 32 more

ruiarodrigues · 2021-12-23T09:25:31Z

I think I will:

change the identity of all Quarkus 1.x schedulers to something different. That way new schedulers will be created as Qaurkus 2.x schedulers
deploy to production (cluster of services)
update the cron as needed and run flyway script to delete the old ones

Do you have another idea/suggestion? Or do you see any problem with this approach?

machi1990 · 2022-01-04T22:43:42Z

I changed the identity of the scheduler from executeCalculationJob to executeCalculationJob_trigger as suggested but quarkus doesn't start.

2021-12-23 08:54:30.509 ERROR [io.qua.dep.dev.IsolatedDevModeMain] (main)  Failed to start quarkus: java.lang.RuntimeException: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.dev.appstate.ApplicationStateNotification.waitForApplicationStart(ApplicationStateNotification.java:51)
	at io.quarkus.runner.bootstrap.StartupActionImpl.runMainClass(StartupActionImpl.java:122)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.firstStart(IsolatedDevModeMain.java:145)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:450)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:67)
	at io.quarkus.bootstrap.app.CuratedApplication.runInCl(CuratedApplication.java:149)
	at io.quarkus.bootstrap.app.CuratedApplication.runInAugmentClassLoader(CuratedApplication.java:105)
	at io.quarkus.deployment.dev.DevModeMain.start(DevModeMain.java:145)
	at io.quarkus.deployment.dev.DevModeMain.main(DevModeMain.java:63)
Caused by: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:104)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:67)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:41)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:120)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:103)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Unable to create Scheduler
	at io.quarkus.quartz.runtime.QuartzScheduler.<init>(QuartzScheduler.java:221)
	at io.quarkus.quartz.runtime.QuartzScheduler_Subclass.<init>(Unknown Source)
	at io.quarkus.quartz.runtime.QuartzScheduler_Bean.create(Unknown Source)
	at io.quarkus.quartz.runtime.QuartzScheduler_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:96)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:29)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:26)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:26)
	at io.quarkus.quartz.runtime.QuartzScheduler_Bean.get(Unknown Source)
	at io.quarkus.quartz.runtime.QuartzScheduler_Bean.get(Unknown Source)
	at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:440)
	at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:453)
	at io.quarkus.arc.impl.ArcContainerImpl.instanceHandle(ArcContainerImpl.java:424)
	at io.quarkus.arc.impl.ArcContainerImpl.instance(ArcContainerImpl.java:233)
	at io.quarkus.arc.runtime.BeanLookupSupplier.get(BeanLookupSupplier.java:29)
	at io.quarkus.vertx.http.runtime.devmode.DevConsoleRecorder.addInfo(DevConsoleRecorder.java:41)
	at io.quarkus.deployment.steps.DevConsoleProcessor$runtimeTemplates-308161071.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.DevConsoleProcessor$runtimeTemplates-308161071.deploy(Unknown Source)
	... 13 more
Caused by: org.quartz.ObjectAlreadyExistsException: Unable to store Trigger with name: 'executeCalculationJob_trigger' and group: 'io.quarkus.scheduler.Scheduler', because one already exists with this identification.
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1184)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$2.executeVoid(JobStoreSupport.java:1068)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3780)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3778)
	at org.quartz.impl.jdbcjobstore.JobStoreCMT.executeInLock(JobStoreCMT.java:245)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1063)
	at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:855)
	at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:249)
	at io.quarkus.quartz.runtime.QuartzScheduler.<init>(QuartzScheduler.java:200)
	... 32 more

I am pretty sure that the identity suggestion works and I've tested it before. The exception you are seeing is rather strange but possibly a different issue worth investigating. Do you mind preparing a reproducer for it and create an issue for it? Thanks

I think I will:

change the identity of all Quarkus 1.x schedulers to something different. That way new schedulers will be created as Qaurkus 2.x schedulers

deploy to production (cluster of services)

update the cron as needed and run flyway script to delete the old ones

Do you have another idea/suggestion?

The other idea was to recreate the quartz table via a database migration using Flyway/Liqubase extensions, since at this point I suspect the corresponding tables might have been corrupted if you did manual modifications.

ruiarodrigues · 2022-01-05T09:52:04Z

Hi!
I didn't do any manual changes on quartz tables. I just change the identity value in the code to the suggested value and this error happened.
Meanwhile I renamed the identity for all our Quartz 1.x schedulers to a different value and they were created correctly and with the new cron expressions. The old ones (xxx_trigger) are still in the quartz tables and I will remove them via a database migration using Flyway.

machi1990 · 2022-01-05T10:19:28Z

Glad you had the issue fixed.

ruiarodrigues · 2022-01-05T13:46:42Z

@machi1990 It would be nice if this could be documented somewhere (maybe in the 2.1 migration guides or somewhere else) for others to be aware of this situation

machi1990 · 2022-02-18T15:48:18Z

@machi1990 It would be nice if this could be documented somewhere (maybe in the 2.1 migration guides or somewhere else) for others to be aware of this situation

We have added a note on 2.1 about the migration.

Furthermore, once this #23634 is merged. The quartz extension will provide a smooth migration for this case.

ruiarodrigues added the kind/bug Something isn't working label Dec 22, 2021

quarkus-bot bot added the area/scheduler label Dec 22, 2021

machi1990 mentioned this issue Feb 18, 2022

Fix NPE when running quartz in database mode and migrating from old versions #23634

Merged

gsmet closed this as completed in #23634 Feb 21, 2022

quarkus-bot bot added this to the 2.8 - main milestone Feb 21, 2022

gsmet modified the milestones: 2.8 - main, 2.7.2.Final Feb 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Quarkus Quartz: Quarkus 1.13.x scheduler is not updated in DB when updated after upgrading to Quarkus 2.x #22479

Quarkus Quartz: Quarkus 1.13.x scheduler is not updated in DB when updated after upgrading to Quarkus 2.x #22479

ruiarodrigues commented Dec 22, 2021

quarkus-bot bot commented Dec 22, 2021

machi1990 commented Dec 23, 2021

mkouba commented Dec 23, 2021

ruiarodrigues commented Dec 23, 2021

ruiarodrigues commented Dec 23, 2021

ruiarodrigues commented Dec 23, 2021

machi1990 commented Jan 4, 2022

ruiarodrigues commented Jan 5, 2022

machi1990 commented Jan 5, 2022

ruiarodrigues commented Jan 5, 2022

machi1990 commented Feb 18, 2022

Quarkus Quartz: Quarkus 1.13.x scheduler is not updated in DB when updated after upgrading to Quarkus 2.x #22479

Quarkus Quartz: Quarkus 1.13.x scheduler is not updated in DB when updated after upgrading to Quarkus 2.x #22479

Comments

ruiarodrigues commented Dec 22, 2021

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

quarkus-bot bot commented Dec 22, 2021

machi1990 commented Dec 23, 2021

mkouba commented Dec 23, 2021

ruiarodrigues commented Dec 23, 2021

ruiarodrigues commented Dec 23, 2021

ruiarodrigues commented Dec 23, 2021

machi1990 commented Jan 4, 2022

ruiarodrigues commented Jan 5, 2022

machi1990 commented Jan 5, 2022

ruiarodrigues commented Jan 5, 2022

machi1990 commented Feb 18, 2022

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)