Standalone JAR packaging of Jenkinsfile Runner is potentially unstable

[oleg-nenashev]

TL;DR: It works by luck and thanks to the Jenkins extension loading failover logic.

META-INF definitions are not merged correctly, and extension annotations from the core might be overridden by plugins. Custom container description handlers are required to handle META-INF/annotations/hudson.Extension though it might be replaced by META-INF/annotations/hudson.Extension.txt. FTR: https://maven.apache.org/plugins/maven-assembly-plugin/examples/single/using-container-descriptor-handlers.html

Sample of an exception which may happen if a new plugin dependency is added:

2020-08-01 19:05:33.802+0000 [id=26]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed Finalizing set up
java.lang.IllegalStateException: Expected 1 instance of jenkins.install.SetupWizard but got 0
        at hudson.ExtensionList.lookupSingleton(ExtensionList.java:451)
        at jenkins.model.Jenkins$16.run(Jenkins.java:3343)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Exception in thread "Executing main" org.jvnet.hudson.reactor.ReactorException: java.lang.IllegalStateException: Expected 1 instance of jenkins.install.SetupWizard but got 0
        at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
        at jenkins.InitReactorRunner.run(InitReactorRunner.java:50)
        at jenkins.model.Jenkins.executeReactor(Jenkins.java:1164)
        at jenkins.model.Jenkins.<init>(Jenkins.java:964)
        at hudson.model.Hudson.<init>(Hudson.java:85)
        at io.jenkins.jenkinsfile.runner.JenkinsEmbedder.newHudson(JenkinsEmbedder.java:323)
        at io.jenkins.jenkinsfile.runner.JenkinsEmbedder.before(JenkinsEmbedder.java:188)
        at io.jenkins.jenkinsfile.runner.JenkinsLauncher.before(JenkinsLauncher.java:111)
        at io.jenkins.jenkinsfile.runner.JenkinsLauncher.launch(JenkinsLauncher.java:75)
        at io.jenkins.jenkinsfile.runner.App.run(App.java:13)
        at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.run(Bootstrap.java:307)
        at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.main(Bootstrap.java:128)
Caused by: java.lang.IllegalStateException: Expected 1 instance of jenkins.install.SetupWizard but got 0
        at hudson.ExtensionList.lookupSingleton(ExtensionList.java:451)
        at jenkins.model.Jenkins$16.run(Jenkins.java:3343)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Dependency order kinda keeps the Jenkins core stable, but at the same time lack of extension cache hits performance significantly in such mode

[oleg-nenashev]

Also there is a glitch in the terminator logic due to missing extensions metadata for modules

2020-12-01 16:40:14.173+0000 [id=72]    WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.slaves.JnlpSlaveAgentProtocol4, annotation=[none]]; skipping this component
java.security.KeyStoreException: JENKINS-41987: no X509Certificate found; perhaps instance-identity module is missing or too old
        at jenkins.slaves.JnlpSlaveAgentProtocol4.<init>(JnlpSlaveAgentProtocol4.java:105)
        at jenkins.slaves.JnlpSlaveAgentProtocol4$$FastClassByGuice$$b470638c.newInstance(<generated>)
        at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
        at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
        at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
        at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:89)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115)
        at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:572)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
Caused: com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.security.KeyStoreException: JENKINS-41987: no X509Certificate found; perhaps instance-identity module is missing or too old
  at jenkins.slaves.JnlpSlaveAgentProtocol4.<init>(JnlpSlaveAgentProtocol4.java:101)

1 error
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
        at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:440)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
        at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:402)
        at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:393)
        at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:346)
        at hudson.ExtensionList.load(ExtensionList.java:381)
        at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
        at hudson.ExtensionList.iterator(ExtensionList.java:172)
        at jenkins.AgentProtocol.of(AgentProtocol.java:112)
        at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:284)

[ecingel-mitek]

I tried pulling jenkins from docker hub today and faced this issue when running locally.
The issue is still open but I wonder if there is any expected resolution in near future or not