Elasticsearch Devservices

docs/src/main/asciidoc/dev-services.adoc

@@ -117,3 +117,11 @@ the server address has not been explicitly configured. More information can be f
 xref:infinispan-client.adoc#dev-services[Infinispan Guide].
 
 include::{generated-dir}/config/quarkus-infinispan-client-infinispan-client-dev-service-build-time-config.adoc[opts=optional, leveloffset=+1]
+
+== Elasticsearch
+
+The Elasticsearch Dev Service will be enabled when one of the Elasticsearch based extensions (Elasticsearch client or Hibernate Search ORM Elasticsearch)
+is present in you application, and the server address has not been explicitly configured.
+More information can be found at the xref:elasticsearch-dev-services.adoc[Elasticsearch Dev Services Guide].
+
+include::{generated-dir}/config/quarkus-elasticsearch-rest-client-common-config-group-elasticsearch-dev-services-build-time-config.adoc[opts=optional, leveloffset=+1]

docs/src/main/asciidoc/elasticsearch-dev-services.adoc

@@ -0,0 +1,68 @@
+////
+This guide is maintained in the main Quarkus repository
+and pull requests should be submitted there:
+https://github.com/quarkusio/quarkus/tree/main/docs/src/main/asciidoc
+////
+= Dev Services for Elasticsearch
+
+include::./attributes.adoc[]
+
+If any Elasticsearch-related extension is present (e.g. `quarkus-elasticsearch-rest-client` or `quarkus-hibernate-search-orm-elasticsearch`),
+Dev Services for Elasticsearch automatically starts an Elasticsearch server in dev mode and when running tests.
+So, you don't have to start a server manually.
+The application is configured automatically.
+
+== Enabling / Disabling Dev Services for Elasticsearch
+
+Dev Services for Elasticsearch is automatically enabled unless:
+
+- `quarkus.devservices.enabled` is set to `false`.
+- `quarkus.elasticsearch.devservices.enabled` is set to `false`
+- the hosts property is configured, depending of the extension used it can be:
+    - `quarkus.elasticsearch.hosts`
+    - `quarkus.hibernate-search-orm.elasticsearch.hosts`
+
+Dev Services for Elasticsearch relies on Docker to start the server.
+If your environment does not support Docker, you will need to start the server manually, or connect to an already running server.
+
+== Shared Elasticsearch
+
+Sometimes you need to share the server between applications.
+Dev Services for Elasticsearch implements a _service discovery_ mechanism for your multiple Quarkus applications running in _dev_ mode to share a single server.
+
+NOTE: Dev Services for Elasticsearch starts the container with the `quarkus-dev-service-elasticsearch` label which is used to identify the container.
+
+If you need multiple (shared) servers, you can configure the `quarkus.elasticsearch.devservices.service-name` attribute and indicate the server name.
+It looks for a container with the same value, or starts a new one if none can be found.
+The default service name is `elasticsearch`.
+
+Sharing is enabled by default in dev mode, but disabled in test mode.
+You can disable the sharing with `quarkus.elasticsearch.devservices.shared=false`.
+
+== Setting the port
+
+By default, Dev Services for Elasticsearch picks a random port and configures the application.
+You can set the port by configuring the `quarkus.elasticsearch.devservices.port` property.
+
+Note that the Elasticsearch hosts property is automatically configured with the chosen port.
+
+== Configuring the image
+
+Dev Services for Elasticsearch only support Elasticsearch based images, Opensearch is not supported at the moment.
+
+If you need to use a different image than the default one you can configure it via
+[source, properties]
+----
+quarkus.elasticsearch.devservices.image-name=docker.elastic.co/elasticsearch/elasticsearch:7.17.0
+----
+
+== Current limitations
+
+Currently, Elasticsearch Dev Services cannot be used if both Hibernate Search ORM Elasticsearch and an Elasticsearch client extension are used,
+this will result in an exception on startup.
+
+Currently, only the default backend for Hibernate Search Elasticsearch is supported, because Dev Services for Elasticsearch can only start one Elasticsearch container.
+
+== Configuration reference
+
+include::{generated-dir}/config/quarkus-elasticsearch-rest-client-common-config-group-elasticsearch-dev-services-build-time-config.adoc[opts=optional, leveloffset=+1]

docs/src/main/asciidoc/elasticsearch.adoc

@@ -261,7 +261,21 @@ quarkus.elasticsearch.hosts = localhost:9200
 
 If you need a more advanced configuration, you can find the comprehensive list of supported configuration properties at the end of this guide.
 
-== Programmatically Configuring Elasticsearch
+[[dev-services]]
+=== Dev Services (Configuration Free Databases)
+Quarkus supports a feature called Dev Services that allows you to start various containers without any config.
+In the case of Elasticsearch this support extends to the default Elasticsearch connection.
+What that means practically is that, if you have not configured `quarkus.elasticsearch.hosts`, Quarkus will automatically
+start an Elasticsearch container when running tests or dev mode, and automatically configure the connection.
+
+When running the production version of the application, the Elasticsearch connection needs to be configured as usual,
+so if you want to include a production database config in your `application.properties` and continue to use Dev Services
+we recommend that you use the `%prod.` profile to define your Elasticsearch settings.
+
+For more information you can read the xref:elasticsearch-dev-services.adoc[Dev Services for Elasticsearch guide].
+
+
+=== Programmatically Configuring Elasticsearch
 On top of the parametric configuration, you can also programmatically apply additional configuration to the client by implementing a `RestClientBuilder.HttpClientConfigCallback` and annotating it with `ElasticsearchClientConfig`. You may provide multiple implementations and configuration provided by each implementation will be applied in a randomly ordered cascading manner.
 
 For example, when accessing an Elasticsearch cluster that is set up for TLS on the HTTP layer, the client needs to trust the certificate that Elasticsearch is using. The following is an example of setting up the client to trust the CA that has signed the certificate that Elasticsearch is using, when that CA certificate is available in a PKCS#12 keystore.

extensions/elasticsearch-rest-client-common/deployment/pom.xml

@@ -25,6 +25,20 @@
             <groupId>io.quarkus</groupId>
             <artifactId>quarkus-elasticsearch-rest-client-common</artifactId>
         </dependency>
+        <dependency>
+            <groupId>io.quarkus</groupId>
+            <artifactId>quarkus-devservices-deployment</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.testcontainers</groupId>
+            <artifactId>elasticsearch</artifactId>
+            <exclusions>
+                <exclusion>
+                    <groupId>junit</groupId>
+                    <artifactId>junit</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
     </dependencies>
 
     <build>

extensions/elasticsearch-rest-client-common/deployment/src/main/java/io/quarkus/elasticsearch/restclient/common/deployment/DevServicesElasticsearchProcessor.java

@@ -0,0 +1,219 @@
+package io.quarkus.elasticsearch.restclient.common.deployment;
+
+import java.time.Duration;
+import java.util.*;
+import java.util.function.Supplier;
+
+import org.jboss.logging.Logger;
+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+import org.testcontainers.utility.DockerImageName;
+
+import io.quarkus.builder.BuildException;
+import io.quarkus.deployment.Feature;
+import io.quarkus.deployment.IsDockerWorking;
+import io.quarkus.deployment.IsNormal;
+import io.quarkus.deployment.annotations.BuildStep;
+import io.quarkus.deployment.builditem.CuratedApplicationShutdownBuildItem;
+import io.quarkus.deployment.builditem.DevServicesResultBuildItem;
+import io.quarkus.deployment.builditem.DevServicesSharedNetworkBuildItem;
+import io.quarkus.deployment.builditem.LaunchModeBuildItem;
+import io.quarkus.deployment.console.ConsoleInstalledBuildItem;
+import io.quarkus.deployment.console.StartupLogCompressor;
+import io.quarkus.deployment.dev.devservices.GlobalDevServicesConfig;
+import io.quarkus.deployment.logging.LoggingSetupBuildItem;
+import io.quarkus.devservices.common.ConfigureUtil;
+import io.quarkus.devservices.common.ContainerAddress;
+import io.quarkus.devservices.common.ContainerLocator;
+import io.quarkus.runtime.configuration.ConfigUtils;
+
+/**
+ * Starts an Elasticsearch server as dev service if needed.
+ */
+public class DevServicesElasticsearchProcessor {
+    private static final Logger log = Logger.getLogger(DevServicesElasticsearchProcessor.class);
+
+    /**
+     * Label to add to shared Dev Service for Elasticsearch running in containers.
+     * This allows other applications to discover the running service and use it instead of starting a new instance.
+     */
+    static final String DEV_SERVICE_LABEL = "quarkus-dev-service-elasticsearch";
+    static final int ELASTICSEARCH_PORT = 9200;
+
+    private static final ContainerLocator elasticsearchContainerLocator = new ContainerLocator(DEV_SERVICE_LABEL,
+            ELASTICSEARCH_PORT);
+
+    static volatile DevServicesResultBuildItem.RunningDevService devService;
+    static volatile ElasticsearchDevServicesBuildTimeConfig cfg;
+    static volatile boolean first = true;
+
+    private final IsDockerWorking isDockerWorking = new IsDockerWorking(true);
+
+    @BuildStep(onlyIfNot = IsNormal.class, onlyIf = GlobalDevServicesConfig.Enabled.class)
+    public DevServicesResultBuildItem startElasticsearchDevService(
+            LaunchModeBuildItem launchMode,
+            ElasticsearchDevServicesBuildTimeConfig configuration,
+            List<DevServicesSharedNetworkBuildItem> devServicesSharedNetworkBuildItem,
+            Optional<ConsoleInstalledBuildItem> consoleInstalledBuildItem,
+            CuratedApplicationShutdownBuildItem closeBuildItem,
+            LoggingSetupBuildItem loggingSetupBuildItem,
+            GlobalDevServicesConfig devServicesConfig,
+            List<DevservicesElasticsearchBuildItem> devservicesElasticsearchBuildItems) throws BuildException {
+
+        if (devservicesElasticsearchBuildItems.isEmpty()) {
+            // safety belt in case a module depends on this one without producing the build item
+            return null;
+        }
+
+        if (devservicesElasticsearchBuildItems.size() > 1) {
+            throw new BuildException(
+                    "Multiple extensions requesting dev services for Elasticsearch found, which is not yet supported." +
+                            "Please de-activate dev services for Elasticsearch using quarkus.elasticsearch.devservices.enabled.",
+                    Collections.emptyList());
+        }
+        DevservicesElasticsearchBuildItem devservicesElasticsearchBuildItem = devservicesElasticsearchBuildItems.get(0);
+
+        if (devService != null) {
+            boolean shouldShutdownTheServer = !configuration.equals(cfg);
+            if (!shouldShutdownTheServer) {
+                return devService.toBuildItem();
+            }
+            shutdownElasticsearch();
+            cfg = null;
+        }
+
+        String hostsConfigProperty = devservicesElasticsearchBuildItem.getHostsConfigProperty();
+        StartupLogCompressor compressor = new StartupLogCompressor(
+                (launchMode.isTest() ? "(test) " : "") + "Elasticsearch Dev Services Starting:",
+                consoleInstalledBuildItem, loggingSetupBuildItem);
+        try {
+            devService = startElasticsearch(configuration, devservicesElasticsearchBuildItem, launchMode,
+                    !devServicesSharedNetworkBuildItem.isEmpty(),
+                    devServicesConfig.timeout);
+            compressor.close();
+        } catch (Throwable t) {
+            compressor.closeAndDumpCaptured();
+            throw new RuntimeException(t);
+        }
+
+        if (devService == null) {
+            return null;
+        }
+
+        // Configure the watch dog
+        if (first) {
+            first = false;
+            Runnable closeTask = () -> {
+                if (devService != null) {
+                    shutdownElasticsearch();
+                }
+                first = true;
+                devService = null;
+                cfg = null;
+            };
+            closeBuildItem.addCloseTask(closeTask, true);
+        }
+        cfg = configuration;
+
+        if (devService.isOwner()) {
+            log.infof(
+                    "Dev Services for Elasticsearch started. Other Quarkus applications in dev mode will find the "
+                            + "server automatically. For Quarkus applications in production mode, you can connect to"
+                            + " this by starting your application with -D%s=%s",
+                    hostsConfigProperty, getElasticsearchHosts(hostsConfigProperty));
+        }
+        return devService.toBuildItem();
+    }
+
+    public static String getElasticsearchHosts(String hostsConfigProperty) {
+        return devService.getConfig().get(hostsConfigProperty);
+    }
+
+    private void shutdownElasticsearch() {
+        if (devService != null) {
+            try {
+                devService.close();
+            } catch (Throwable e) {
+                log.error("Failed to stop the Elasticsearch server", e);
+            } finally {
+                devService = null;
+            }
+        }
+    }
+
+    private DevServicesResultBuildItem.RunningDevService startElasticsearch(ElasticsearchDevServicesBuildTimeConfig config,
+            DevservicesElasticsearchBuildItem devservicesElasticsearchBuildItem,
+            LaunchModeBuildItem launchMode, boolean useSharedNetwork, Optional<Duration> timeout) throws BuildException {
+        if (!config.enabled.orElse(true)) {
+            // explicitly disabled
+            log.debug("Not starting dev services for Elasticsearch, as it has been disabled in the config.");
+            return null;
+        }
+
+        String hostsConfigProperty = devservicesElasticsearchBuildItem.getHostsConfigProperty();
+        // Check if elasticsearch hosts property is set
+        if (ConfigUtils.isPropertyPresent(hostsConfigProperty)) {
+            log.debugf("Not starting dev services for Elasticsearch, the %s property is configured.", hostsConfigProperty);
+            return null;
+        }
+
+        if (!isDockerWorking.getAsBoolean()) {
+            log.warnf(
+                    "Docker isn't working, please configure the Elasticsearch hosts property (%s).", hostsConfigProperty);
+            return null;
+        }
+
+        // We only support ELASTIC container for now
+        if (devservicesElasticsearchBuildItem.getDistribution() == DevservicesElasticsearchBuildItem.Distribution.OPENSEARCH) {
+            throw new BuildException("Dev services for Elasticsearch didn't support Opensearch", Collections.emptyList());
+        }
+
+        // Hibernate search Elasticsearch have a version configuration property, we need to check that it is coherent
+        // with the image we are about to launch
+        if (devservicesElasticsearchBuildItem.getVersion() != null) {
+            String containerTag = config.imageName.substring(config.imageName.indexOf(':') + 1);
+            if (!containerTag.startsWith(devservicesElasticsearchBuildItem.getVersion())) {
+                throw new BuildException(
+                        "Dev services for Elasticsearch detected a version mismatch, container image is " + config.imageName
+                                + " but the configured version is " + devservicesElasticsearchBuildItem.getVersion() +
+                                ". Either configure a different image or disable dev services for Elasticsearch.",
+                        Collections.emptyList());
+            }
+        }
+
+        final Optional<ContainerAddress> maybeContainerAddress = elasticsearchContainerLocator.locateContainer(
+                config.serviceName,
+                config.shared,
+                launchMode.getLaunchMode());
+
+        // Starting the server
+        final Supplier<DevServicesResultBuildItem.RunningDevService> defaultElasticsearchSupplier = () -> {
+            ElasticsearchContainer container = new ElasticsearchContainer(
+                    DockerImageName.parse(config.imageName));
+            ConfigureUtil.configureSharedNetwork(container, "elasticsearch");
+            if (config.serviceName != null) {
+                container.withLabel(DEV_SERVICE_LABEL, config.serviceName);
+            }
+            if (config.port.isPresent()) {
+                container.setPortBindings(List.of(config.port.get() + ":" + config.port.get()));
+            }
+            timeout.ifPresent(container::withStartupTimeout);
+            container.addEnv("ES_JAVA_OPTS", config.javaOpts);
+            // Disable security as else we would need to configure it correctly to avoid tons of WARNING in the log
+            container.addEnv("xpack.security.enabled", "false");
+
+            container.start();
+            return new DevServicesResultBuildItem.RunningDevService(Feature.ELASTICSEARCH_REST_CLIENT_COMMON.getName(),
+                    container.getContainerId(),
+                    container::close,
+                    hostsConfigProperty, container.getHttpHostAddress());
+        };
+
+        return maybeContainerAddress
+                .map(containerAddress -> new DevServicesResultBuildItem.RunningDevService(
+                        Feature.ELASTICSEARCH_REST_CLIENT_COMMON.getName(),
+                        containerAddress.getId(),
+                        null,
+                        hostsConfigProperty, containerAddress.getUrl()))
+                .orElseGet(defaultElasticsearchSupplier);
+    }
+}

extensions/elasticsearch-rest-client-common/deployment/src/main/java/io/quarkus/elasticsearch/restclient/common/deployment/DevservicesElasticsearchBuildItem.java

@@ -0,0 +1,39 @@
+package io.quarkus.elasticsearch.restclient.common.deployment;
+
+import io.quarkus.builder.item.MultiBuildItem;
+
+public final class DevservicesElasticsearchBuildItem extends MultiBuildItem {
+    private final String hostsConfigProperty;
+
+    private final String version;
+    private final Distribution distribution;
+
+    public DevservicesElasticsearchBuildItem(String configItemName) {
+        this.hostsConfigProperty = configItemName;
+        this.version = null;
+        this.distribution = Distribution.ELASTIC;
+    }
+
+    public DevservicesElasticsearchBuildItem(String configItemName, String version, Distribution distribution) {
+        this.hostsConfigProperty = configItemName;
+        this.version = version;
+        this.distribution = distribution;
+    }
+
+    public String getHostsConfigProperty() {
+        return hostsConfigProperty;
+    }
+
+    public String getVersion() {
+        return version;
+    }
+
+    public Distribution getDistribution() {
+        return distribution;
+    }
+
+    public enum Distribution {
+        ELASTIC,
+        OPENSEARCH
+    }
+}