Updates to initialization, docs

integration-tests-ex/README.md

@@ -21,7 +21,49 @@
 
 This directory builds a Docker image for Druid, then uses that image, along
 with test configuration to run tests. This version greatly evolves the
-integration tests from the earlier form. See the last section for details.
+integration tests from the earlier form. See the [History](docs/history.md)
+section for details.
+
+## Shortcuts
+
+See [Quickstart](docs/quickstart.md) for an explanation.
+
+### Build Druid
+
+```bash
+mvn clean package -P dist,skip-static-checks,skip-tests -Dmaven.javadoc.skip=true -T1.0C
+```
+
+### Build the Test Image
+
+```bash
+cd $DRUID_DEV/integration-tests-ex/it-image
+mvn -P test-image install
+```
+
+### Run an IT from the Command Line
+
+```bash
+mvn install -P docker-tests -pl :<group>
+```
+
+Or
+
+```bash
+cd $DRUID_DEV/integration-tests-ex/<group>
+mvn verify -P docker-tests
+```
+
+### Run an IT from the IDE
+
+Start the cluster:
+
+```bash
+cd $DRUID_DEV/integration-tests-ex/<group>
+./cluster.sh up
+```
+
+Then launch the test as a JUnit test.
 
 ## Contents
 
@@ -35,6 +77,7 @@ integration tests from the earlier form. See the last section for details.
 * [Test configuration](docs/test-config.md)
 * [Test structure](docs/tests.md)
 * [Test runtime semantics](docs/runtime.md)
+* [Scripts](docs/scripts.md)
 * [Dependencies](docs/dependencies.md)
 * [Debugging](docs/debugging.md)
 
@@ -53,7 +96,7 @@ The goal of the present version is to simplify development.
   Maven and reside in the local build cache.)
 * Use official images for dependencies to avoid the need to
   download, install, and manage those dependencies.
-* Ensure that it is easy to manually build the image, launch
+* Make it is easy to manually build the image, launch
   a cluster, and run a test against the cluster.
 * Convert tests to JUnit so that they will easily run in your
   favorite IDE, just like other Druid tests.
@@ -76,13 +119,3 @@ feature is:
 * Create a normal unit test and run it to verify your code.
 * Create an integration test that double-checks the code in
   a live cluster.
-
-The result should also speed up Travis builds because a single
-Maven run can produce the Druid artifacts and run all integration tests.
-
-We can have the option to run the integration tests using different
-settings: one with the MariaDB connector, say, another with MySQL.
-Each of these might be a separate Travis run, since the tests run
-with different configurations (as given by a Maven profile). But,
-we won't have to run a separte Travis job for each integration test
-group.

integration-tests-ex/do-cluster.sh

@@ -0,0 +1,95 @@
+#! /bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#--------------------------------------------------------------------
+
+# Helper script that does the actual work of launching the cluster.
+# A script in each module sets the module-specific context, the
+# TEST_DIR.
+
+#set -x
+
+if [ -z "$TEST_DIR" ]; then
+	echo "This is an internal script. Run cluster.sh in each module."
+	exit 1
+fi
+
+# 'up' command by default, else whatever is the argument
+CMD='up'
+if [ $# -ge 1 ]; then
+	CMD=$1
+fi
+
+# All commands need env vars.
+
+ENV_FILE=$TEST_DIR/../it-image/target/env.sh
+if [ ! -f $ENV_FILE ]; then
+	echo "Please build the Docker test image before testing" 1>&2
+	exit 1
+fi
+
+source $ENV_FILE
+
+export SHARED_DIR=$TEST_DIR/target/shared
+
+# Permissions in some build setups are screwed up. See above. The user
+# which runs Docker does not have permission to write into the /shared
+# directory. Force ownership to allow writing.
+chmod -R a+rwx ${IT_MODULE}/target/shared
+
+# Dummies just to get the compose files to shut up
+# TODO: Remove after conversion
+export OVERRIDE_ENV=
+export DRUID_INTEGRATION_TEST_GROUP=dummy
+
+# Print environment for debugging
+#env
+
+# Dump lots of information to debug Docker failures when run inside
+# of a build environment where we can't inspect Docker directly.
+function show_status {
+	echo "===================================="
+	ls -l target/shared
+	echo "docker ps -a"
+	docker ps -a
+	# Was: --filter status=exited
+	for id in $(docker ps -a --format "{{.ID}}"); do
+	    echo "===================================="
+	    echo "Logs for Container ID $id"
+		docker logs $id | tail -n 20
+	done
+	echo "===================================="
+}
+
+cd $TEST_DIR/druid-cluster
+if [ "$CMD" == 'up' ]; then
+	# Must start with an empty DB to keep MySQL happy
+	rm -rf $SHARED_DIR/db
+	mkdir -p $SHARED_DIR/logs
+	mkdir -p $SHARED_DIR/tasklogs
+	mkdir -p $SHARED_DIR/db
+	mkdir -p $SHARED_DIR/kafka
+	mkdir -p $SHARED_DIR/resources
+	cp $TEST_DIR/../assets/log4j2.xml $SHARED_DIR/resources
+	chmod -R a+rwx $SHARED_DIR
+	docker-compose up -d
+	#show_status
+elif [ "$CMD" == 'status' ]; then
+	show_status
+else
+	#show_status
+	docker-compose $CMD
+fi

integration-tests-ex/docs/maven.md

@@ -33,17 +33,15 @@ through Maven will build Druid and run the tests.
 Use the following command to run the ITs, assuming `DRUID_DEV` points
 to your Druid development directory:
 
-
 ```bash
 cd $DRUID_DEV
-mvn clean install -P dist,test-image,docker-tests \
-    -P skip-static-checks -Ddruid.console.skip=true \
+mvn clean package -P dist,test-image,docker-tests,skip-static-checks \
     -Dmaven.javadoc.skip=true -DskipUTs=true
 ```
 
 The various pieces are:
 
-* `clean`: Remove any existing artifacts, and the existing Docker image.
+* `clean`: Remove any existing artifacts, and any existing Docker image.
 * `install`: Build the Druid code and write it to the local Maven repo.
 * `-P dist`: Create the Druid distribution tarball by pulling jars from
   the local Maven repo.
@@ -52,15 +50,15 @@ The various pieces are:
   for Docker.
 * `-P docker-tests`: Run the new ITs.
 * Everything else: ignore parts of the build not needed for the ITs, such
-  as static checks, unit tests, the console, Javadoc, etc.
+  as static checks, unit tests, Javadoc, etc.
 
 Once you've done the above once, you can do just the specific part you want
 to repeat during development. See below for details.
 
 ### Skipping Tests
 
 The revised integration tests use the [Maven failsafe plugin]
-(https://maven.apache.org/surefire/maven-failsafe-plugin/) which shared
+(https://maven.apache.org/surefire/maven-failsafe-plugin/) which shares
 code with the [Maven Surefire plugin](
 https://maven.apache.org/surefire/maven-surefire-plugin/)
 used to run unit tests. One shared item is the `skipTests` flag.
@@ -90,7 +88,8 @@ The IT process is a collection of sub-modules:
 At present, the following test groups are fully or partly converted:
 
 * `it-high-availability`: Fully converted (there is only one test file.)
-* `it-batch-indexer`: Partially converted.
+* `it-batch-indexer`: Fully converted.
+* `it-input-format`: Fully converted.
 
 The "it-" prefix ensures that the test-related projects are grouped
 together in your IDE.
@@ -99,8 +98,8 @@ together in your IDE.
 
 It turns out that, for obscure reasons, we must use a "flat" module
 structure under the root Druid `pom.xml` even though the modules
-themselves are in folders within `docker-tests`. That is, we cannot
-create a `docker-tests` Maven module to hold the IT modules. The
+themselves are in folders within `integration-tests-ex`. That is, we cannot
+create a `integration-tests-ex` Maven module to hold the IT modules. The
 reason has to do with the fact that Maven has no good way to
 obtain the directory that contains the root `pom.xml` file. Yet,
 we need this directory to find configuration files for some of the
@@ -112,11 +111,16 @@ goal directly from the `mvn` command line: Maven happily ignores the
 By using a two-level module structure, we can punt and just always
 assume that `${project.parent.basedir}` points to the root Druid
 directory. More than you wanted to know, but now you know why there
-is no `docker-tests` module as there should be.
+is no `integration-tests-ex` module as there should be.
 
 As a result, you may have to hunt in your IDE to find the non-project
-files in the `docker-tests` directory. Look in the root `druid`
-project, in the `docker-tests` folder.
+files in the `integration-tests-ex` directory. Look in the root `druid`
+project, in the `integration-tests-ex` folder.
+
+Another very unfortunate side effect is that we must include a great
+deal of copy/past in each IT module, since we can't pull these dependencies
+into an IT base `pom.xml` file. If you're looking for a challenge, sorting
+out this situation is a good candidate!
 
 ## `org.apache.druid.testing2`
 
@@ -128,7 +132,7 @@ places. The goal is to merge the `testing2` package back into
 
 ## Profiles
 
-Integration test artifacts are build only if you specifically request them
+Integration test artifacts are built only if you specifically request them
 using a profile.
 
 * `-P test-image` builds the test Docker image.
@@ -161,12 +165,15 @@ If you add additional dependencies, please follow the above process. See the
 
 ## Environment Variables
 
+The build environment users environment variables to pass information to Maven.
 Maven communicates with Docker and Docker Compose via environment variables
 set in the `exec-maven-plugin` of various `pom.xml` files. The environment
 variables then flow into either the Docker build script (`Dockerfile`) or the
 various Docker Compose scripts (`docker-compose.yaml`). It can be tedious to follow
 this flow. A quick outline:
 
+* The build environment (such as Travis) sets environment variables, or passes values
+  to maven via the `-d<var>=<value` syntax.
 * Maven, via `exec-maven-plugin`, sets environment variables typically from Maven's
   own variables for things like versions.
 * `exec-maven-plugin` invokes a script to do the needes shell fiddling. The environment
@@ -175,7 +182,7 @@ this flow. A quick outline:
 * When building, the script passes the environment variables to Docker as build
   arguments.
 * `Dockerfile` typically passes the build arguments to the image as environment
-  variables as the same name.
+  variables of the same name.
 * When running, the script passes environment variables implicitly to Docker Compose,
   which uses them in the various service configurations and/or environment variable
   settings passed to each container.
@@ -190,7 +197,7 @@ configuration:
 The easiest way to test is to insert (or enable, or view) the environment in the
 image:
 
-```shell
+```bash
 env
 ```
 
@@ -212,9 +219,27 @@ mvn install -P test-image
 Here "install" means to install the image in the local Docker image repository.
 
 Even with the above, Druid's Maven configuration has many plugins which Maven
-must grind through on each
-build if you are impatient, you can actually remove Maven from the build path for
-the image and tests by using a script to do what `exec-maven-plugin` does. You just
-have to be sure to set the needed environment variables to the proper values.
-Once you do that, creating an image, or launching a cluster, is quite fast.
+must grind through on each build. A faster route is to `cd` into the target
+directory, so Maven only runs that one project:
+
+```bash
+cd $DRUID_DEV/integration-tests-ex/it-image
+mvn -P test-image install
+```
+
+If you truely impatient, you can actually remove Maven from the build path for
+the image and tests by using a script to do what `exec-maven-plugin` does. Let
+Maven do its thing once which will create the `integration-tests-ex/target/env.sh`
+file that contains the required envionment variables. After that, you can rerun
+the image or the tests by reusing that file.
+
+For the image:
+
+```bash
+cd $DRUID_DEV/integration-tests-ex/it-image
+./rebuild.sh
+```
+
+See the [Quickstart](docs/quickstart.md) for how to run tests this way.
+Using this trick, creating an image, or launching a cluster, is quite fast.
 See [the Docker section](docker.md) for details.