Quarkus ignores all "quarkus.*" environment properties on Kubernetes #36359

fedinskiy · 2023-10-09T14:47:45Z

Describe the bug

I have an application, which uses consul, I use property "quarkus.consul-config.agent.host-port" to provide the application with path to deployed consul instance. Starting from 6 or 7 of October, when I deploy this app on Openshift using 999-SNAPSHOT build, the app can not retrieve this property and fails to start.
The property is stored in openshift DeploymentConfig as "template.spec.containers.env.name".
If the property is put into property file, or the property variable inside the yaml is renamed into "QUARKUS_CONSUL_CONFIG_AGENT_HOST_PORT", then everything works fine.

Expected behavior

Application starts succefully

Actual behavior

Application fails with the following error:

16:07:48,407 INFO  [app] java.lang.RuntimeException: Failed to start quarkus
16:07:48,408 INFO  [app] 	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
16:07:48,408 INFO  [app] 	at io.quarkus.runtime.Application.start(Application.java:101)
16:07:48,409 INFO  [app] 	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
16:07:48,409 INFO  [app] 	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
16:07:48,409 INFO  [app] 	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
16:07:48,410 INFO  [app] 	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
16:07:48,410 INFO  [app] 	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
16:07:48,411 INFO  [app] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:07:48,411 INFO  [app] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
16:07:48,411 INFO  [app] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:07:48,412 INFO  [app] 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
16:07:48,412 INFO  [app] 	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:61)
16:07:48,413 INFO  [app] 	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:32)
16:07:48,413 INFO  [app] Caused by: java.lang.RuntimeException: An error occurred while attempting to fetch configuration from Consul.
16:07:48,413 INFO  [app] 	at io.quarkus.consul.config.runtime.ConsulConfigSourceFactory.getConfigSources(ConsulConfigSourceFactory.java:82)
16:07:48,414 INFO  [app] 	at io.quarkus.consul.config.runtime.ConsulConfigSourceFactory.getConfigSources(ConsulConfigSourceFactory.java:28)
16:07:48,414 INFO  [app] 	at io.quarkus.consul.config.runtime.ConsulConfigSourceFactory.getConfigSources(ConsulConfigSourceFactory.java:19)
16:07:48,415 INFO  [app] 	at io.smallrye.config.ConfigSourceFactory$ConfigurableConfigSourceFactory.getConfigSources(ConfigSourceFactory.java:58)
16:07:48,415 INFO  [app] 	at io.smallrye.config.ConfigurableConfigSource.getConfigSources(ConfigurableConfigSource.java:50)
16:07:48,416 INFO  [app] 	at io.smallrye.config.SmallRyeConfig$ConfigSources.mapLateSources(SmallRyeConfig.java:687)
16:07:48,416 INFO  [app] 	at io.smallrye.config.SmallRyeConfig$ConfigSources.<init>(SmallRyeConfig.java:577)
16:07:48,416 INFO  [app] 	at io.smallrye.config.SmallRyeConfig.<init>(SmallRyeConfig.java:68)
16:07:48,417 INFO  [app] 	at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:698)
16:07:48,417 INFO  [app] 	at io.quarkus.runtime.generated.Config.readConfig(Unknown Source)
16:07:48,417 INFO  [app] 	at io.quarkus.runtime.generated.Config.createRunTimeConfig(Unknown Source)
16:07:48,418 INFO  [app] 	at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source)
16:07:48,418 INFO  [app] 	... 13 more
16:07:48,419 INFO  [app] Caused by: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8500
16:07:48,419 INFO  [app] 	at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:79)
16:07:48,420 INFO  [app] 	at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
16:07:48,420 INFO  [app] 	at io.quarkus.consul.config.runtime.ConsulConfigSourceFactory.getConfigSources(ConsulConfigSourceFactory.java:79)
16:07:48,420 INFO  [app] 	... 24 more
16:07:48,421 INFO  [app] Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8500
16:07:48,422 INFO  [app] Caused by: java.net.ConnectException: Connection refused
16:07:48,422 INFO  [app] 	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
16:07:48,422 INFO  [app] 	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
16:07:48,423 INFO  [app] 	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
16:07:48,423 INFO  [app] 	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
16:07:48,424 INFO  [app] 	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
16:07:48,424 INFO  [app] 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
16:07:48,425 INFO  [app] 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
16:07:48,425 INFO  [app] 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
16:07:48,425 INFO  [app] 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
16:07:48,426 INFO  [app] 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
16:07:48,426 INFO  [app] 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
16:07:48,427 INFO  [app] 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
16:07:48,427 INFO  [app] 	at java.base/java.lang.Thread.run(Thread.java:833)

How to Reproduce?

git clone [email protected]:quarkus-qe/quarkus-test-framework.git
mvn clean install -Pframework
Log in into openshift
This fails: " mvn clean install -Pexamples,openshift -pl examples/consul/ -Dit.test=OpenShiftGreetingResourceIT -Dquarkus.platform.version=999-SNAPSHOT"
This works as expected: mvn clean install -Pexamples,openshift -pl examples/consul/ -Dit.test=OpenShiftGreetingResourceIT -Dquarkus.platform.version=3.4.2

If option -Dts.global.delete.folder.on.exit=false is added, then yaml file examples/consul/target/OpenShiftGreetingResourceIT/app/openshift.yml contains deployment config
By default, the project is cleared afterwards, add -Dts.openshift.delete.project.after.all=false -Dts.openshift.ephemeral.namespaces.enabled=false to use the persistent one

Output of `uname -a` or `ver`

6.4.14-200.fc38.x86_64

Output of `java -version`

17.0.5, vendor: GraalVM Community

GraalVM version (if different from Java)

No response

Quarkus version or git rev

6be99a4

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)

Additional information

No response

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2023-10-09T14:47:48Z

/cc @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift)

fedinskiy · 2023-10-09T14:51:13Z

Probably also area/config

fedinskiy · 2023-10-09T15:13:07Z

Commit 39381f9 is not affected, so I presume this is caused by #35322

geoand · 2023-10-09T15:14:43Z

cc @radcortez

radcortez · 2023-10-09T17:11:47Z

I'll have a look.

fedinskiy · 2023-10-10T15:52:24Z

Addition: this affect all "quarkus.*" properties, not just "consul-config.agent.host-port"

radcortez · 2023-10-11T22:03:33Z

I have an application, which uses consul, I use property "quarkus.consul-config.agent.host-port" to provide the application with path to deployed consul instance. Starting from 6 or 7 of October, when I deploy this app on Openshift using 999-SNAPSHOT build, the app can not retrieve this property and fails to start.

Are you using precisely this property name? quarkus.consul-config.agent.host-port, or is it the environment variable name?

The property is stored in openshift DeploymentConfig as "template.spec.containers.env.name".

Are you setting environment variable names in dotted format?

If the property is put into property file, or the property variable inside the yaml is renamed into "QUARKUS_CONSUL_CONFIG_AGENT_HOST_PORT", then everything works fine.

From the last comment, I'm guessing that you are using quarkus.consul-config.agent.host-port in a source that should only take Environment Variables as QUARKUS_CONSUL_CONFIG_AGENT_HOST_PORT. It may have worked in the past, but this was never intended to work...

fedinskiy · 2023-10-12T06:19:38Z

@radcortez yes, the app uses environment variables in dotted format and for some reason this worked (presumably for a long time).
I agree, that this is strange, but we should probably document, that it doesn't work anymore.

radcortez · 2023-10-12T09:35:53Z

Sure. Feel free to document it. Thanks!

fedinskiy · 2023-10-12T09:57:55Z

That should go to release notes I suppose. Who is handling that for Quarkus and how do I contact that person?

Before that change we used quarkus CLI properties as environment variables in openshift manifests (which we should not have do) and for some reason it worked (which it should not have do). When this behaviour was changed upstream, our jobs started to fail, this change should fix that. See quarkusio/quarkus#36359 for details

radcortez · 2023-10-12T10:01:27Z

Release notes are added here:
https://github.com/quarkusio/quarkus/wiki/Migration-Guide-3.5

Are you able to edit and add the notes in the Config section?

Before that change we used quarkus CLI properties as environment variables in openshift manifests (which we should not have do) and for some reason it worked (which it should not have do). When this behaviour was changed upstream, our jobs started to fail, this change should fix that. See quarkusio/quarkus#36359 for details

fedinskiy · 2023-10-12T10:52:50Z

No, I can not.

I prepared a note, could you add it into the guide?

System properties can not be used as environmental variables anymore.

For some time, it was possible to have quarkus environment variables in dotted.lower-case (style, used for system properties) and such variables were read and accepted by Quarkus. This was accidental and against specifications[1], and was removed in this release. From now on, Quarkus accepts environment variables only in SCREAMING_SNAKE_CASE.

[1] https://quarkus.io/guides/config-reference#system-properties

janderssonse · 2023-11-11T07:13:36Z

This helped me, we have quite a few config envs in the x.y.z-format to change to X_Y_Z now. While the release notes makes it quite clear, @radcortez @fedinskiy last note should really be a added to the https://quarkus.io/guides/config-reference#environment-variables documentation paragraph, as it stil gives examples of envrionment naming resolutions in x.Y.Z-formats. And so, people that didn't read every release note will contiinue trying that format. @radcortez Should I open a seperate issue regadring environment documentation?

radcortez · 2023-11-11T13:18:31Z

This helped me, we have tons of config envs in the x.y.z-format to change to X_Y_Z now.

You may want to hold that change :)

I was not aware that so many users were relying on this, so I reaaded the support in #36917.

janderssonse · 2023-11-11T13:41:26Z

Thanks @radcortez , I have now read the thread at #36917 , and that docker handling with dot mixed upper/lower case mode was surely interesting, I can see why you would want to add the dot-support /mixedmode:) We already started the posix compatible journey as we are deploying that version atm, (no big issue for us)

radcortez · 2023-11-13T11:18:27Z

Great :)

fedinskiy added the kind/bug Something isn't working label Oct 9, 2023

quarkus-bot bot added the area/kubernetes label Oct 9, 2023

geoand added the area/config label Oct 9, 2023

fedinskiy changed the title ~~Quarkus ignores environment property "consul-config.agent.host-port" on Kubernetes~~ Quarkus ignores all "quarkus.*" environment properties on Kubernetes Oct 10, 2023

fedinskiy mentioned this issue Oct 12, 2023

Fix nightly failures quarkus-qe/quarkus-test-framework#904

Merged

10 tasks

fedinskiy mentioned this issue Oct 18, 2023

Since 3.5.0 AMQ hostname cannot be overriden with environment variable in OpenShift, previously it could be #36549

Closed

radcortez closed this as completed Oct 30, 2023

radcortez added this to the 3.5.0 milestone Oct 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Quarkus ignores all "quarkus.*" environment properties on Kubernetes #36359

Quarkus ignores all "quarkus.*" environment properties on Kubernetes #36359

fedinskiy commented Oct 9, 2023

quarkus-bot bot commented Oct 9, 2023

fedinskiy commented Oct 9, 2023

fedinskiy commented Oct 9, 2023

geoand commented Oct 9, 2023

radcortez commented Oct 9, 2023

fedinskiy commented Oct 10, 2023

radcortez commented Oct 11, 2023

fedinskiy commented Oct 12, 2023

radcortez commented Oct 12, 2023

fedinskiy commented Oct 12, 2023

radcortez commented Oct 12, 2023

fedinskiy commented Oct 12, 2023

janderssonse commented Nov 11, 2023 •

edited

Loading

radcortez commented Nov 11, 2023

janderssonse commented Nov 11, 2023 •

edited

Loading

radcortez commented Nov 13, 2023

Quarkus ignores all "quarkus.*" environment properties on Kubernetes #36359

Quarkus ignores all "quarkus.*" environment properties on Kubernetes #36359

Comments

fedinskiy commented Oct 9, 2023

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

quarkus-bot bot commented Oct 9, 2023

fedinskiy commented Oct 9, 2023

fedinskiy commented Oct 9, 2023

geoand commented Oct 9, 2023

radcortez commented Oct 9, 2023

fedinskiy commented Oct 10, 2023

radcortez commented Oct 11, 2023

fedinskiy commented Oct 12, 2023

radcortez commented Oct 12, 2023

fedinskiy commented Oct 12, 2023

radcortez commented Oct 12, 2023

fedinskiy commented Oct 12, 2023

janderssonse commented Nov 11, 2023 • edited Loading

radcortez commented Nov 11, 2023

janderssonse commented Nov 11, 2023 • edited Loading

radcortez commented Nov 13, 2023

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

janderssonse commented Nov 11, 2023 •

edited

Loading

janderssonse commented Nov 11, 2023 •

edited

Loading