Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamodb - URL connection http client support #4471

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions bom/runtime/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -2230,6 +2230,11 @@
<artifactId>apache-client</artifactId>
<version>${awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>url-connection-client</artifactId>
<version>${awssdk.version}</version>
</dependency>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What annoys me a bit is that you get all the dependencies with the extension. I wonder if it's a wise choice and we shouldn't be more careful about that especially in the lambda use case.

That being said, I'm not totally sure we can find a nice way to do that properly but IMHO it's worth a try.

If we can get to the point of the dependencies being optional and we can throw a warning or an error if something is configured for a "backend" and the correct dependencies are not in the classpath, that would be OK IMHO.

I don't think it's blocking for this PR tbh so I think we could merge this and see how we can improve things later.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea and in fact tried it, check the latest commits. In general:

  1. I made client deps as optional
  2. the deployment processor checks if required service exists on the classpath, if not throws deploymentException
  3. the application should explicitly add required dependency to the classpath

Do you think it make sense ? Once you verify the idea, I'll update docs & quickstart


<dependency>
<groupId>com.microsoft.azure.functions</groupId>
Expand Down
54 changes: 48 additions & 6 deletions docs/src/main/asciidoc/dynamodb-guide.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Keep in mind it's actively developed and does not support yet all the features a

The Quarkus extension supports two programming models:

* Blocking access using the Apache HTTP Client
* Blocking access using URL Connection HTTP client (by default) or the Apache HTTP Client
* https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/basics-async.html[Asynchronous programming] based on JDK's `CompletableFuture` objects and the Netty HTTP client.

In this guide, we see how you can get your REST services to use the DynamoDB locally and on AWS.
Expand Down Expand Up @@ -318,6 +318,38 @@ The implementation is pretty straightforward and you just need to define your en
== Configuring DynamoDB clients

Both DynamoDB clients (sync and async) are configurable via the `application.properties` file that can be provided in the `src/main/resources` directory.
Additionally, you need to added to the classpath a proper implementation of the sync client. By default the extension uses URL connection HTTP client, so
you need to add a URL connection client dependency to the `pom.xml` file:

[source,xml]
----
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>url-connection-client</artifactId>
</dependency>
----

If you want to use Apache HTTP client instead, configure it as follows:
[source,properties]
----
quarkus.dynamodb.sync-client.type=apache
----

And add following dependency to the application `pom.xml`:
[source,xml]
----
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>apache-client</artifactId>
<exclusions>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
----
NOTE: It's important to exclude `commons-logging` from the client dependency to force Apache HTTP client to use Quarkus logger.

If you're going to use a local DynamoDB instance, configure it as follows:

Expand All @@ -326,24 +358,24 @@ If you're going to use a local DynamoDB instance, configure it as follows:
quarkus.dynamodb.endpoint-override=http://localhost:8000

quarkus.dynamodb.aws.region=eu-central-1
quarkus.dynamodb.aws.credentials.type=STATIC
quarkus.dynamodb.aws.credentials.type=static
quarkus.dynamodb.aws.credentials.static-provider.access-key-id=test-key
quarkus.dynamodb.aws.credentials.static-provider.secret-access-key=test-secret
----

- `quarkus.dynamodb.aws.region` - It's required by the client, but since you're using a local DynamoDB instance you can pick any valid AWS region.
- `quarkus.dynamodb.aws.credentials.type` - Set `STATIC` credentials provider with any values for `access-key-id` and `secret-access-key`
- `quarkus.dynamodb.aws.credentials.type` - Set `static` credentials provider with any values for `access-key-id` and `secret-access-key`
- `quarkus.dynamodb.endpoint-override` - Override the DynamoDB client to use a local instance instead of an AWS service

If you want to work with an AWS account, you'd need to set it with:
[source,properties]
----
quarkus.dynamodb.aws.region=<YOUR_REGION>
quarkus.dynamodb.aws.credentials.type=DEFAULT
quarkus.dynamodb.aws.credentials.type=default
----

- `quarkus.dynamodb.aws.region` you should set it to the region where you provisioned the DynamoDB table,
- `quarkus.dynamodb.aws.credentials.type` - use the `DEFAULT` credentials provider chain that looks for credentials in this order:
- `quarkus.dynamodb.aws.credentials.type` - use the `default` credentials provider chain that looks for credentials in this order:
- Java System Properties - `aws.accessKeyId` and `aws.secretKey`
* Environment Variables - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
* Credential profiles file at the default location (`~/.aws/credentials`) shared by all AWS SDKs and the AWS CLI
Expand Down Expand Up @@ -400,7 +432,7 @@ public class FruitAsyncService extends AbstractService {
}
----

And an asynchronous REST resource:
Create an asynchronous REST resource:

[source,java]
----
Expand Down Expand Up @@ -445,6 +477,16 @@ public class FruitAsyncResource {
}
----

And add Netty HTTP client dependency to the `pom.xml`:

[source,xml]
----
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>netty-nio-client</artifactId>
</dependency>
----

== Configuration Reference

include::{generated-dir}/config/quarkus-dynamodb.adoc[opts=optional, leveloffset=+1]
Expand Down
21 changes: 21 additions & 0 deletions extensions/amazon-dynamodb/deployment/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,27 @@
<artifactId>rest-assured</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>netty-nio-client</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>url-connection-client</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>apache-client</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>

<build>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
import java.util.List;
import java.util.stream.Collectors;

import javax.enterprise.inject.spi.DeploymentException;

import org.jboss.jandex.DotName;
import org.jboss.jandex.Type;

Expand All @@ -27,26 +29,29 @@
import io.quarkus.deployment.builditem.substrate.SubstrateProxyDefinitionBuildItem;
import io.quarkus.deployment.builditem.substrate.SubstrateResourceBuildItem;
import io.quarkus.deployment.configuration.ConfigurationError;
import io.quarkus.dynamodb.runtime.ApacheHttpClientConfig;
import io.quarkus.dynamodb.runtime.AwsCredentialsProviderType;
import io.quarkus.dynamodb.runtime.DynamodbClientProducer;
import io.quarkus.dynamodb.runtime.DynamodbConfig;
import io.quarkus.dynamodb.runtime.DynamodbRecorder;
import io.quarkus.dynamodb.runtime.NettyHttpClientConfig;
import io.quarkus.dynamodb.runtime.SyncHttpClientConfig;
import io.quarkus.dynamodb.runtime.SyncHttpClientConfig.SyncClientType;
import io.quarkus.dynamodb.runtime.TlsManagersProviderConfig;
import io.quarkus.dynamodb.runtime.TlsManagersProviderType;
import software.amazon.awssdk.core.interceptor.ExecutionInterceptor;
import software.amazon.awssdk.http.SdkHttpService;
import software.amazon.awssdk.http.apache.ApacheSdkHttpService;
import software.amazon.awssdk.http.async.SdkAsyncHttpService;
import software.amazon.awssdk.http.nio.netty.NettySdkAsyncHttpService;
import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.utils.StringUtils;

public class DynamodbProcessor {
public static final String AWS_SDK_APPLICATION_ARCHIVE_MARKERS = "software/amazon/awssdk";

private static final String APACHE_HTTP_SERVICE = "software.amazon.awssdk.http.apache.ApacheSdkHttpService";
private static final String NETTY_HTTP_SERVICE = "software.amazon.awssdk.http.nio.netty.NettySdkAsyncHttpService";
private static final String URL_HTTP_SERVICE = "software.amazon.awssdk.http.urlconnection.UrlConnectionSdkHttpService";

private static final List<String> INTERCEPTOR_PATHS = Arrays.asList(
"software/amazon/awssdk/global/handlers/execution.interceptors",
"software/amazon/awssdk/services/dynamodb/execution.interceptors");
Expand Down Expand Up @@ -112,26 +117,40 @@ DynamodbClientBuildItem analyzeDynamodbClientInjectionPoints(BeanRegistrationPha
}

if (createSyncClient) {
//Register Apache client as sync client
proxyDefinition
.produce(new SubstrateProxyDefinitionBuildItem("org.apache.http.conn.HttpClientConnectionManager",
"org.apache.http.pool.ConnPoolControl",
"software.amazon.awssdk.http.apache.internal.conn.Wrapped"));

serviceProvider.produce(
new ServiceProviderBuildItem(SdkHttpService.class.getName(), ApacheSdkHttpService.class.getName()));
if (config.syncClient.type == SyncClientType.APACHE) {
checkClasspath(APACHE_HTTP_SERVICE, "apache-client");

//Register Apache client as sync client
proxyDefinition.produce(
new SubstrateProxyDefinitionBuildItem("org.apache.http.conn.HttpClientConnectionManager",
"org.apache.http.pool.ConnPoolControl",
"software.amazon.awssdk.http.apache.internal.conn.Wrapped"));

serviceProvider.produce(new ServiceProviderBuildItem(SdkHttpService.class.getName(), APACHE_HTTP_SERVICE));
} else {
checkClasspath(URL_HTTP_SERVICE, "url-connection-client");
serviceProvider.produce(new ServiceProviderBuildItem(SdkHttpService.class.getName(), URL_HTTP_SERVICE));
}
}

if (createAsyncClient) {
checkClasspath(NETTY_HTTP_SERVICE, "netty-nio-client");
//Register netty as async client
serviceProvider.produce(
new ServiceProviderBuildItem(SdkAsyncHttpService.class.getName(),
NettySdkAsyncHttpService.class.getName()));
serviceProvider.produce(new ServiceProviderBuildItem(SdkAsyncHttpService.class.getName(), NETTY_HTTP_SERVICE));
}

return new DynamodbClientBuildItem(createSyncClient, createAsyncClient);
}

private void checkClasspath(String className, String dependencyName) {
try {
Class.forName(className, true, Thread.currentThread().getContextClassLoader());
} catch (ClassNotFoundException e) {
throw new DeploymentException(
"Missing 'software.amazon.awssdk:" + dependencyName + "' dependency on the classpath");
}
}

@BuildStep
@Record(ExecutionTime.RUNTIME_INIT)
void buildClients(DynamodbClientBuildItem clientBuildItem, DynamodbRecorder recorder,
Expand Down Expand Up @@ -193,17 +212,19 @@ private static void checkConfig(DynamodbConfig config, List<String> knownInterce
}
}

private static void checkSyncClientConfig(ApacheHttpClientConfig syncClient) {
if (syncClient.maxConnections <= 0) {
throw new ConfigurationError("quarkus.dynamodb.sync-client.max-connections may not be negative or zero.");
}
if (syncClient.proxy != null && syncClient.proxy.enabled) {
URI proxyEndpoint = syncClient.proxy.endpoint;
if (proxyEndpoint != null) {
validateProxyEndpoint(proxyEndpoint, "sync");
private static void checkSyncClientConfig(SyncHttpClientConfig syncClient) {
if (syncClient.type == SyncClientType.APACHE) {
if (syncClient.apache.maxConnections <= 0) {
throw new ConfigurationError("quarkus.dynamodb.sync-client.max-connections may not be negative or zero.");
}
if (syncClient.apache.proxy != null && syncClient.apache.proxy.enabled) {
URI proxyEndpoint = syncClient.apache.proxy.endpoint;
if (proxyEndpoint != null) {
validateProxyEndpoint(proxyEndpoint, "sync");
}
}
validateTlsManagersProvider(syncClient.apache.tlsManagersProvider, "sync");
}
validateTlsManagersProvider(syncClient.tlsManagersProvider, "sync");
}

private static void checkAsyncClientConfig(NettyHttpClientConfig asyncClient) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
import io.quarkus.test.QuarkusUnitTest;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;

public class DynamodbSyncClientBrokenConfigTest {
public class DynamodbSyncApacheClientBrokenConfigTest {

@Inject
DynamoDbClient client;
Expand All @@ -21,7 +21,7 @@ public class DynamodbSyncClientBrokenConfigTest {
static final QuarkusUnitTest config = new QuarkusUnitTest()
.setExpectedException(ConfigurationError.class)
.setArchiveProducer(() -> ShrinkWrap.create(JavaArchive.class)
.addAsResource("sync-broken-config.properties", "application.properties"));
.addAsResource("sync-apache-broken-config.properties", "application.properties"));

@Test
public void test() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
import io.quarkus.test.QuarkusUnitTest;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;

public class DynamodbSyncClientBrokenProxyConfigTest {
public class DynamodbSyncApacheClientBrokenProxyConfigTest {

@Inject
DynamoDbClient client;
Expand All @@ -21,7 +21,7 @@ public class DynamodbSyncClientBrokenProxyConfigTest {
static final QuarkusUnitTest config = new QuarkusUnitTest()
.setExpectedException(ConfigurationError.class)
.setArchiveProducer(() -> ShrinkWrap.create(JavaArchive.class)
.addAsResource("sync-broken-proxy-config.properties", "application.properties"));
.addAsResource("sync-apache-broken-proxy-config.properties", "application.properties"));

@Test
public void test() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@
import io.quarkus.test.QuarkusUnitTest;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;

public class DynamodbSyncClientFullConfigTest {
public class DynamodbSyncApacheClientFullConfigTest {

@Inject
DynamoDbClient client;

@RegisterExtension
static final QuarkusUnitTest config = new QuarkusUnitTest()
.setArchiveProducer(() -> ShrinkWrap.create(JavaArchive.class)
.addAsResource("sync-full-config.properties", "application.properties"));
.addAsResource("sync-apache-full-config.properties", "application.properties"));

@Test
public void test() {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package io.quarkus.dynamodb.deployment;

import javax.inject.Inject;

import org.jboss.shrinkwrap.api.ShrinkWrap;
import org.jboss.shrinkwrap.api.spec.JavaArchive;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;

import io.quarkus.test.QuarkusUnitTest;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;

public class DynamodbSyncUrlConnClientFullConfigTest {

@Inject
DynamoDbClient client;

@RegisterExtension
static final QuarkusUnitTest config = new QuarkusUnitTest()
.setArchiveProducer(() -> ShrinkWrap.create(JavaArchive.class)
.addAsResource("sync-urlconn-full-config.properties", "application.properties"));

@Test
public void test() {
// Application should start with full config.
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ quarkus.dynamodb.enable-endpoint-discovery=false
quarkus.dynamodb.endpoint-override=http://localhost:8000

quarkus.dynamodb.aws.region=us-east-1
quarkus.dynamodb.aws.credentials.type=STATIC
quarkus.dynamodb.aws.credentials.type=static
quarkus.dynamodb.aws.credentials.static-provider.access-key-id=test-key
quarkus.dynamodb.aws.credentials.static-provider.secret-access-key=test-secret

Expand All @@ -15,13 +15,13 @@ quarkus.dynamodb.async-client.connection-acquisition-timeout=0.01S
quarkus.dynamodb.async-client.connection-time-to-live=0.01S
quarkus.dynamodb.async-client.connection-max-idle-time=0.01S
quarkus.dynamodb.async-client.use-idle-connection-reaper=false
quarkus.dynamodb.async-client.protocol = HTTP1_1
quarkus.dynamodb.async-client.protocol = http1-1
quarkus.dynamodb.async-client.max-http2-streams = 10
quarkus.dynamodb.async-client.ssl-provider = JDK
quarkus.dynamodb.async-client.ssl-provider = jdk
quarkus.dynamodb.async-client.event-loop.override = true
quarkus.dynamodb.async-client.event-loop.number-of-threads = 5
quarkus.dynamodb.async-client.event-loop.thread-name-prefix = Quarkus-Netty-EventLoop-
quarkus.dynamodb.async-client.proxy.enabled = true
quarkus.dynamodb.async-client.proxy.endpoint = http://127.1.1.1
quarkus.dynamodb.async-client.proxy.non-proxy-hosts = localhost, hostlocal
quarkus.dynamodb.async-client.tls-managers-provider.type=SYSTEM_PROPERTY
quarkus.dynamodb.async-client.tls-managers-provider.type=system-property
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
quarkus.dynamodb.aws.region=us-east-1

quarkus.dynamodb.async-client.tls-managers-provider.type=FILE_STORE
quarkus.dynamodb.async-client.tls-managers-provider.type=file-store
quarkus.dynamodb.async-client.tls-managers-provider.file-store.path=/tmp/file.key
quarkus.dynamodb.async-client.tls-managers-provider.file-store.type=pkcs11
quarkus.dynamodb.async-client.tls-managers-provider.file-store.password=thePassword
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
quarkus.dynamodb.aws.region=us-east-2

quarkus.dynamodb.aws.credentials.type=DEFAULT
quarkus.dynamodb.aws.credentials.type=default
quarkus.dynamodb.aws.credentials.default-provider.async-credential-update-enabled=true
quarkus.dynamodb.aws.credentials.default-provider.reuse-last-provider-enabled=true

Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
quarkus.dynamodb.aws.credentials.type=PROCESS
quarkus.dynamodb.aws.credentials.type=process

Loading