Missing an HTTP endpoint to ensure the Presto Docker Container is ready

[linghengqian]

Expected Behavior or Use Case

• Missing an HTTP endpoint to ensure the Presto Docker Container is ready. This caused a bit of trouble in the use of testscontainers-java. https://prestodb.io/docs/current/installation/deploy-docker.html mentions that you only need to check if ======= SERVER STARTED ======== is output. But this still seems insufficient to indicate that the Presto Docker Container is ready. I wrote a simple unit test at https://github.com/linghengqian/testcontainers-presto-health-check-test . Just execute the following command to verify. Execute the following command on the Ubuntu 22.04.4 LTS instance with SDKMAN! and Docker.

sdk install java 17.0.11-ms
sdk use java 17.0.11-ms

git clone [email protected]:linghengqian/testcontainers-presto-health-check-test.git
cd ./testcontainers-presto-health-check-test/
./mvnw clean test

• Logically, this unit test is very simple.

@Testcontainers
public class PrestoTest {

    @Container
    public GenericContainer<?> container = new GenericContainer<>(DockerImageName.parse("prestodb/presto:0.288"))
            .withCopyFileToContainer(
                    MountableFile.forClasspathResource("default", Transferable.DEFAULT_DIR_MODE),
                    "/opt/presto-server/etc"
            )
            .withExposedPorts(8080)
            .waitingFor(
                    new LogMessageWaitStrategy().withRegEx(".*======== SERVER STARTED ========.*")
                            .withStartupTimeout(Duration.of(60, ChronoUnit.SECONDS))
            );

    @Test
    void test() throws SQLException {
        Properties props = new Properties();
        props.setProperty("user", "test");
        try (Connection connection = DriverManager.getConnection("jdbc:presto://" + container.getHost() + ":" + container.getMappedPort(8080) + "/", props);
             Statement statement = connection.createStatement()) {
            statement.execute("""
                    CREATE TABLE memory.default.table_with_array AS SELECT 1 id, ARRAY[1, 42, 2, 42, 4, 42] my_array
                    """);
            try (ResultSet resultSet = statement.executeQuery(
                    """
                            SELECT nationkey, element
                            FROM tpch.tiny.nation
                            JOIN memory.default.table_with_array twa ON nationkey = twa.id
                            CROSS JOIN UNNEST(my_array) a(element)
                            ORDER BY element OFFSET 1 FETCH FIRST 3 ROWS ONLY
                            """
            )
            ) {
                List<Integer> actualElements = new ArrayList<>();
                while (resultSet.next()) {
                    actualElements.add(resultSet.getInt("element"));
                }
                assertThat(actualElements, is(Arrays.asList(2, 4, 42)));
            }
        }
    }
}

• This unit test can easily trigger the Error Log of com.facebook.presto.spi.PrestoException: No nodes available to run query. Although sometimes this unit test can be executed successfully.

java.sql.SQLException: Query failed (#20240716_105708_00000_xzrms): No nodes available to run query
        at com.facebook.presto.jdbc.PrestoResultSet.resultsException(PrestoResultSet.java:1841)

• It would be nice if there was a simple HTTP endpoint that could be used to replace the following logic to verify that the Presto Server is running.

.waitingFor(
                    new LogMessageWaitStrategy().withRegEx(".*======== SERVER STARTED ========.*")
                            .withStartupTimeout(Duration.of(60, ChronoUnit.SECONDS))
            )

Presto Component, Service, or Connector

• Presto Component.

Possible Implementation

• To be honest I'm not sure.

Example Screenshots (if appropriate):

• Normally this wouldn't require any screenshots.

Context

• This relates to the findings from Change the Docker Image used by the Presto module to prestodb/presto testcontainers/testcontainers-java#8946. Earlier investigation came from Improve GraalVM Reachability Metadata and corresponding nativeTest related unit tests apache/shardingsphere#29052 .

[tdcmeehan]

Perhaps we can use the /v1/info/state endpoint (wait for it to return ACTIVE), and set the cluster.required-workers-active property to >= 1 (default is 0). Instead of waiting for SERVER STARTED, we can wait for the endpoint to return the desired state.

[linghengqian]

Perhaps we can use the /v1/info/state endpoint (wait for it to return ACTIVE), and set the cluster.required-workers-active property to >= 1 (default is 0). Instead of waiting for SERVER STARTED, we can wait for the endpoint to return the desired state.

• @tdcmeehan Hi, I updated the unit test and the corresponding Log at linghengqian/testcontainers-presto-health-check-test@3ea8f8c . But it seems that the following changes lead to another Error, namely Presto server is still initializing.

.waitingFor(
                    new HttpWaitStrategy()
                            .forPath("/v1/info/state")
                            .forPort(8080)
                            .forResponsePredicate("\"ACTIVE\""::equals)
            );

• And /opt/presto-server/etc/config.properties.

coordinator=true
node-scheduler.include-coordinator=true
http-server.http.port=8080
discovery-server.enabled=true
discovery.uri=http://localhost:8080
cluster.required-workers-active=1
offset-clause-enabled=true

• The log stack doesn't show anything.

java.sql.SQLException: Query failed (#20240716_152749_00000_pf2mr): Presto server is still initializing
        at com.facebook.presto.jdbc.PrestoResultSet.resultsException(PrestoResultSet.java:1841)

[lutongzero]

It looks like you can try calling the /v1/cluster endpoint. This endpoint returns an example like {"runningQueries":1,"blockedQueries":0,"queuedQueries":0,"activeWorkers":99"}. You can determine if the cluster is ready by checking activeWorkers > 0.

[linghengqian]

It looks like you can try calling the /v1/cluster endpoint. This endpoint returns an example like {"runningQueries":1,"blockedQueries":0,"queuedQueries":0,"activeWorkers":99"}. You can determine if the cluster is ready by checking activeWorkers > 0.

• @lutongzero Hi, I made the following changes in linghengqian/testcontainers-presto-health-check-test@5030324 .

.waitingFor(
                    new HttpWaitStrategy()
                            .forPath("/v1/cluster")
                            .forPort(8080)
                            .forResponsePredicate(anObject -> {
                                try {
                                    return new ObjectMapper()
                                            .readTree(anObject)
                                            .get("activeWorkers")
                                            .asInt() > 0;
                                } catch (IOException e) {
                                    throw new RuntimeException(e);
                                }
                            })
            );

• But judging only this JSON entry will still result in Error Log Presto server is still initializing.

java.sql.SQLException: Query failed (#20240717_030349_00000_nnbf6): Presto server is still initializing
        at com.facebook.presto.jdbc.PrestoResultSet.resultsException(PrestoResultSet.java:1841)

[linghengqian]

• So I checked com.facebook.presto.nativeworker.TestPrestoContainerBasicQueries of [Native] Add Functional Testing Framework using Testcontainers #23087, and the only way to verify Presto Docker Containers is to let the main thread sleep for 5 seconds?

coordinator.start();
worker.start();

// Wait for worker to announce itself.
TimeUnit.SECONDS.sleep(5);

[lutongzero]

It looks like this issue has been fixed. Please see this PR #23585.

[linghengqian]

• Late reply, I confirmed in linghengqian/testcontainers-presto-health-check-test@0908e9d that this issue has been fixed since Presto 0.290.
• I have also updated Change the Docker Image used by the Presto module to prestodb/presto testcontainers/testcontainers-java#8946 .
• Should I close the current issue?

[linghengqian]

• I did some testing on Add GraalVM Reachability Metadata and corresponding nativeTest for Presto apache/shardingsphere#34429 and I realized that Fix Server Active State Endpoint #23585 doesn't actually fix the /v1/info/state bug.
• Even https://github.com/linghengqian/testcontainers-presto-health-check-test , there is a configuration like this,

@Container
    public GenericContainer<?> container = new GenericContainer<>(DockerImageName.parse("prestodb/presto:0.290"))
            .withCopyFileToContainer(
                    MountableFile.forClasspathResource("default", Transferable.DEFAULT_DIR_MODE),
                    "/opt/presto-server/etc"
            )
            .withExposedPorts(8080)
            .waitingFor(
                    new WaitAllStrategy()
                            .withStrategy(Wait.forLogMessage(".*======== SERVER STARTED ========.*", 1))
                            .withStrategy(Wait.forHttp("/v1/info/state").forPort(8080).forResponsePredicate("\"ACTIVE\""::equals))
            );

coordinator=true
node-scheduler.include-coordinator=true
http-server.http.port=8080
discovery-server.enabled=true
discovery.uri=http://localhost:8080
cluster.required-workers-active=1
cluster.required-resource-managers-active=1
cluster.required-coordinators-active=1
offset-clause-enabled=true

• The Error Log of No nodes available to run query is still appearing randomly. I described this phenomenon at linghengqian/testcontainers-presto-health-check-test@dfa1ba2 . This unstable Error Log phenomenon is also affecting Add GraalVM Reachability Metadata and corresponding nativeTest for Presto apache/shardingsphere#34429 and Change the Docker Image used by the Presto module to prestodb/presto testcontainers/testcontainers-java#8946 's CI.

$./mvnw clean test
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.93 s <<< FAILURE! -- in com.lingh.PrestoTest
[ERROR] com.lingh.PrestoTest.test -- Time elapsed: 11.81 s <<< ERROR!
java.sql.SQLException: Query failed (#20250123_041210_00000_9z2if): No nodes available to run query
        at com.facebook.presto.jdbc.PrestoResultSet.resultsException(PrestoResultSet.java:1841)
        at com.facebook.presto.jdbc.PrestoResultSet$ResultsPageIterator.computeNext(PrestoResultSet.java:1821)
        at com.facebook.presto.jdbc.PrestoResultSet$ResultsPageIterator.computeNext(PrestoResultSet.java:1760)
        at com.facebook.presto.jdbc.internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
        at com.facebook.presto.jdbc.internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
        at com.facebook.presto.jdbc.internal.guava.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
        at com.facebook.presto.jdbc.internal.guava.collect.Iterators$ConcatenatedIterator.getTopMetaIterator(Iterators.java:1379)
        at com.facebook.presto.jdbc.internal.guava.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1395)
        at com.facebook.presto.jdbc.PrestoResultSet.next(PrestoResultSet.java:146)
        at com.facebook.presto.jdbc.PrestoStatement.internalExecute(PrestoStatement.java:291)
        at com.facebook.presto.jdbc.PrestoStatement.execute(PrestoStatement.java:230)
        at com.lingh.PrestoTest.test(PrestoTest.java:49)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: com.facebook.presto.spi.PrestoException: No nodes available to run query
        at com.facebook.presto.spi.NodeManager.getRequiredWorkerNodes(NodeManager.java:36)
        at com.facebook.presto.plugin.memory.MemoryMetadata.beginCreateTable(MemoryMetadata.java:217)
        at com.facebook.presto.plugin.memory.MemoryMetadata.beginCreateTable(MemoryMetadata.java:71)
        at com.facebook.presto.metadata.MetadataManager.beginCreateTable(MetadataManager.java:846)
        at com.facebook.presto.execution.scheduler.TableWriteInfo.createWriterTarget(TableWriteInfo.java:96)
        at com.facebook.presto.execution.scheduler.TableWriteInfo.createWriterTarget(TableWriteInfo.java:118)
        at com.facebook.presto.execution.scheduler.TableWriteInfo.createTableWriteInfo(TableWriteInfo.java:76)
        at com.facebook.presto.execution.scheduler.SectionExecutionFactory.createSectionExecutions(SectionExecutionFactory.java:175)
        at com.facebook.presto.execution.scheduler.SqlQueryScheduler.createStageExecutions(SqlQueryScheduler.java:357)
        at com.facebook.presto.execution.scheduler.SqlQueryScheduler.<init>(SqlQueryScheduler.java:246)
        at com.facebook.presto.execution.scheduler.SqlQueryScheduler.createSqlQueryScheduler(SqlQueryScheduler.java:176)
        at com.facebook.presto.execution.SqlQueryExecution.planDistribution(SqlQueryExecution.java:657)
        at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:485)
        at com.facebook.presto.$gen.Presto_0_290_6a04267____20250123_041206_1.run(Unknown Source)
        at com.facebook.presto.execution.SqlQueryManager.createQuery(SqlQueryManager.java:319)
        at com.facebook.presto.dispatcher.LocalDispatchQuery.lambda$startExecution$8(LocalDispatchQuery.java:213)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   PrestoTest.test:49 » SQL Query failed (#20250123_041210_00000_9z2if): No nodes available to run query
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  13.809 s
[INFO] Finished at: 2025-01-23T12:12:11+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.2.5:test (default-test) on project testcontainers-presto-health-check-test: 
[ERROR] 
[ERROR] Please refer to /home/linghengqian/TwinklingLiftWorks/git/public/testcontainers-presto-health-check-test/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException