from version 1.11.0 Vault MicroProfile Config Source stopped working #14707

hernael · 2021-01-29T15:31:21Z

Describe the bug
from version 1.11.0 Vault MicroProfile Config Source stopped working to use vault with Databases (Vault MicroProfile Config Source) and OIDC

Expected behavior
quarkus fetched vault kv values first and then try verificted dependent properties. In version 1.10.5 works fine, but in 1.11.0 (and 1.11.1) does not work

Actual behavior
on startup throw error:

ERROR [io.qua.dep.dev.IsolatedDevModeMain] (main) Failed to start quarkus: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
	[error]: Build step io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor#config threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value password in property quarkus.datasource.password
	at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
	at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
	at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
	at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
	at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
	at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:192)
	at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:149)
	at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:209)
	at io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor.config(ConfigEditorProcessor.java:55)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
	at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
	at java.base/java.lang.Thread.run(Thread.java:829)
	at org.jboss.threads.JBossThread.run(JBossThread.java:501)

To Reproduce
in a new application in version 1.11.0 or 1.11.1 add vault, quarkus-reactive-pg-client, quarkus-resteasy, quarkus-resteasy-mutiny, quarkus-oidc, quarkus-vertx dependencies in appplication.properties add the following properties:

Configuration

quarkus.vertx.prefer-native-transport=true
quarkus.http.so-reuse-port=true
quarkus.http.tcp-quick-ack=true
quarkus.http.tcp-cork=true
quarkus.http.tcp-fast-open=true
quarkus.thread-pool.core-threads=5

quarkus.vault.url=http://localhost:8201
quarkus.vault.authentication.client-token=x.xxxxxxxxxxxxxx
quarkus.vault.secret-config-kv-path.database=database-product/credentials
quarkus.vault.secret-config-kv-path.oidc=identity-server/oidc

quarkus.datasource.db-kind=postgresql
quarkus.datasource.username=${database.username}
quarkus.datasource.password=${database.password}
quarkus.datasource.reactive.url=${database.url}
quarkus.datasource.reactive.max-size=20
quarkus.hibernate-orm.database.generation=none

quarkus.oidc.enabled=true
quarkus.oidc.application-type=service
quarkus.oidc.discovery-enabled=true
quarkus.oidc.auth-server-url=${oidc.auth-server-url}
quarkus.oidc.client-id=${oidc.client-id}

Environment (please complete the following information):

Output of uname -a or ver: Linux 5.3.18-lp152.60-default x86_64 x86_64 x86_64 GNU/Linux
Output of java -version: OpenJDK 64-Bit Server VM Corretto-11.0.10.9.1 (build 11.0.10+9-LTS, mixed mode)
GraalVM version (if different from Java):
Quarkus version or git rev: 1.11.0 and 1.11.1
Build tool (ie. output of mvnw --version or gradlew --version): 3.6.3

The text was updated successfully, but these errors were encountered:

ghost · 2021-01-29T15:31:24Z

/cc @vsevel

vsevel · 2021-01-29T20:16:51Z

do you have the logs by any chance?

hernael · 2021-01-29T21:25:29Z

yes thanks! the full log is:

[INFO] Scanning for projects...
[INFO] 
[INFO] ------------< com.example.template:template-core >-------------
[INFO] Building template-core 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- quarkus-maven-plugin:1.11.1.Final:dev (default-cli) @ template-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 68 source files to /home/hernael/template/template-core/target/classes
Listening for transport dt_socket at address: 5005
2021-01-29 17:18:48,310 INFO  [org.jbo.threads] (main) JBoss Threads version 3.2.0.Final
2021-01-29 17:18:48,416 ERROR [io.qua.dep.dev.IsolatedDevModeMain] (main) Failed to start quarkus: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
	[error]: Build step io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor#config threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value database.url in property quarkus.datasource.reactive.url
	at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
	at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
	at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
	at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
	at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
	at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:192)
	at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:149)
	at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:209)
	at io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor.config(ConfigEditorProcessor.java:55)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
	at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
	at java.base/java.lang.Thread.run(Thread.java:829)
	at org.jboss.threads.JBossThread.run(JBossThread.java:501)

	at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:308)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:182)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:59)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.firstStart(IsolatedDevModeMain.java:78)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:372)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:55)
	at io.quarkus.bootstrap.app.CuratedApplication.runInCl(CuratedApplication.java:128)
	at io.quarkus.bootstrap.app.CuratedApplication.runInAugmentClassLoader(CuratedApplication.java:81)
	at io.quarkus.deployment.dev.DevModeMain.start(DevModeMain.java:144)
	at io.quarkus.deployment.dev.DevModeMain.main(DevModeMain.java:63)
Caused by: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
	[error]: Build step io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor#config threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value database.url in property quarkus.datasource.reactive.url
	at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
	at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
	at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
	at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
	at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
	at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:192)
	at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:149)
	at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:209)
	at io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor.config(ConfigEditorProcessor.java:55)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
	at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
	at java.base/java.lang.Thread.run(Thread.java:829)
	at org.jboss.threads.JBossThread.run(JBossThread.java:501)

	at io.quarkus.builder.Execution.run(Execution.java:116)
	at io.quarkus.builder.BuildExecutionBuilder.execute(BuildExecutionBuilder.java:79)
	at io.quarkus.deployment.QuarkusAugmentor.run(QuarkusAugmentor.java:153)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:306)
	... 9 more
Caused by: java.util.NoSuchElementException: SRCFG00011: Could not expand value database.url in property quarkus.datasource.reactive.url
	at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
	at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
	at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
	at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
	at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
	at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:192)
	at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:149)
	at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:209)
	at io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor.config(ConfigEditorProcessor.java:55)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
	at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
	at java.base/java.lang.Thread.run(Thread.java:829)
	at org.jboss.threads.JBossThread.run(JBossThread.java:501)

2021-01-29 17:18:48,416 INFO  [io.qua.dep.dev.IsolatedDevModeMain] (main) Attempting to start hot replacement endpoint to recover from previous Quarkus startup failure

vsevel · 2021-01-29T21:38:41Z

not sure. and I am not familiar with the dev mode. the vault extension should print some logs. I guess it fails to run before we try to fetch ${database.url}, which should have been filled by the vault config source.
it does not look like an issue with the vault extension itself, but rather that it is not being called.
@hernael is the application running fine in normal mode (i.e. not in dev mode)
@gsmet any idea who we should direct that issue to?

hernael · 2021-01-29T21:49:48Z

@vsevel But the strangest thing about all this is that it work fine with version 1.10.5, but just changing to 1.11.0 fails

vsevel · 2021-01-29T21:53:06Z

is the application running fine in normal mode (i.e. not in dev mode)?

hernael · 2021-01-29T23:20:55Z

ok, i could already run it in normal mode!
And yes it works fine

vsevel · 2021-01-30T08:36:42Z

so it is probably related to how dev mode is initializing the app. one difference is that the vault extension was moved to the boostrap phase in 1.11 #13498. this might be the reason why it was still working in 1.10.

hernael · 2021-01-30T20:31:27Z

all this means that somebody have to move the vault extension in next versions?

gsmet · 2021-02-01T09:00:33Z

@radcortez could you try to help @vsevel figure this out?

radcortez · 2021-02-01T11:56:37Z

sure, I'll have a look

radcortez · 2021-02-02T02:10:29Z

The issue is actually related with the Dev Console. Since the Config Dev Console is done at build time, the Vault ConfigSource is not available at that point, and the expansion fails.

I've added a fix in #14309 to move the Dev Console Config to Runtime. This approach is also more correct, since it displays all values in the runtime configs and not only build time.

Until the fix is not applied, as a workaround, you can set an empty default value with ${database.url:} (notice the : in the expansion). If you don't want to mess with the main properties, just set them in the dev profile: %dev.quarkus.datasource.reactive.url=${database.url:}.

hernael · 2021-02-02T03:20:52Z

thank you for the alternative solution Roberto!

gsmet · 2021-02-02T09:30:22Z

The issue is actually related with the Dev Console. Since the Config Dev Console is done at build time, the Vault ConfigSource is not available at that point, and the expansion fails.

I'm a bit worried about this. Won't we slow down dev mode significantly if we start doing all sorts of things with the Dev UI before it's even accessed?

/cc @stuartwdouglas

radcortez · 2021-02-02T10:17:33Z

I'm a bit worried about this. Won't we slow down dev mode significantly if we start doing all sorts of things with the Dev UI before it's even accessed?

In this case, I think we are fine. The configuration has to be loaded anyway for the application. Potentially, we may have a slowdown if a ConfigSource does not cache the results and we are doing double lookups (for the console and for the actual application config).

Lets not forget, that currently the values we see in the console may not be correct, since they only take account the build sources, a value may be overridden in a runtime source we won't see that reflected in the console.

vsevel · 2021-02-02T12:53:02Z

and the issue certainly applies to all config sources that are fetching properties at runtime, which is then very limiting.

hernael added the kind/bug Something isn't working label Jan 29, 2021

ghost added the area/vault label Jan 29, 2021

ebullient mentioned this issue Feb 1, 2021

HashiCorp key vault does not override application.properties #14743

Closed

radcortez mentioned this issue Feb 2, 2021

Improve Config Console #14309

Merged

stuartwdouglas closed this as completed in #14309 Feb 11, 2021

ghost added this to the 1.13 - master milestone Feb 11, 2021

gsmet modified the milestones: 1.13 - master, 1.11.3.Final Feb 11, 2021

hernael mentioned this issue Mar 30, 2021

from version 1.13.0 Vault MicroProfile Config Source stopped working #16123

Closed

hernael mentioned this issue Oct 27, 2021

in version 2.3.1 and 2.4.0 Vault MicroProfile Config Source stopped working with OIDC #21047

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

from version 1.11.0 Vault MicroProfile Config Source stopped working #14707

from version 1.11.0 Vault MicroProfile Config Source stopped working #14707

hernael commented Jan 29, 2021

ghost commented Jan 29, 2021

vsevel commented Jan 29, 2021

hernael commented Jan 29, 2021

vsevel commented Jan 29, 2021

hernael commented Jan 29, 2021

vsevel commented Jan 29, 2021

hernael commented Jan 29, 2021

vsevel commented Jan 30, 2021

hernael commented Jan 30, 2021

gsmet commented Feb 1, 2021

radcortez commented Feb 1, 2021

radcortez commented Feb 2, 2021

hernael commented Feb 2, 2021

gsmet commented Feb 2, 2021

radcortez commented Feb 2, 2021

vsevel commented Feb 2, 2021

from version 1.11.0 Vault MicroProfile Config Source stopped working #14707

from version 1.11.0 Vault MicroProfile Config Source stopped working #14707

Comments

hernael commented Jan 29, 2021

ghost commented Jan 29, 2021

vsevel commented Jan 29, 2021

hernael commented Jan 29, 2021

vsevel commented Jan 29, 2021

hernael commented Jan 29, 2021

vsevel commented Jan 29, 2021

hernael commented Jan 29, 2021

vsevel commented Jan 30, 2021

hernael commented Jan 30, 2021

gsmet commented Feb 1, 2021

radcortez commented Feb 1, 2021

radcortez commented Feb 2, 2021

hernael commented Feb 2, 2021

gsmet commented Feb 2, 2021

radcortez commented Feb 2, 2021

vsevel commented Feb 2, 2021