PHOENIX-5827 Add a maven repo to PQS and optionally bundle phoenix-cl…

…ient * Move the cluster.xml to the prescribed location per Maven convention. * Use maven-assembly-plugin to create a Maven repo in the assembly * Pull the phoenix-client jar from central and localize so PQS can actually function * Fix the phoenix-client jar name so that it's picked up by phoenix_utils.py * Prevent old Jetty versions from sneaking in on HBase 1.x * Build the ServerCustomizer and expose configuration to enable it Move integration tests to a dedicated module to avoid jetty clashing With Hadoop2/HBase1, we have to deal with conflicting versions of Jetty at runtime. Avatica's ServerCustomizers expose an unshaded Jetty class (Server), which causes problems when we have Jetty6 also on the classpath. We can avoid this by moving all things that touch Avatica's version of Jetty into queryserver, shade that, and then only invoke HBase/Hadoop from within a different module (letting their Jetty6 run wild). The only gripe is that BasicAuthenticationServerCustomizer has to live in src/main/java to get shaded, even though it is test-only code. This is a minor grievance for what's a horrible runtime solution. This all gets much better with Hadoop3/HBase2 where Jetty is shaded. Closes apache#25 Signed-off-by: Istvan Toth <[email protected]>
arenadata · Apr 22, 2020 · ea5c806 · ea5c806
1 parent ac320cf
commit ea5c806
Show file tree

Hide file tree

Showing 30 changed files with 781 additions and 210 deletions.
diff --git a/.github/workflows/maven.yml b/.github/workflows/maven.yml
@@ -13,4 +13,4 @@ jobs:
       with:
         java-version: 1.8
     - name: Build with Maven
-      run: mvn clean install
+      run: mvn -B clean install
diff --git a/.gitignore b/.gitignore
@@ -2,6 +2,7 @@
 *.class
 *.war
 *.jar
+dependency-reduced-pom.xml
 
 # python
 *.pyc
@@ -26,4 +27,4 @@ target/
 release/
 RESULTS/
 CSV_EXPORT/
-.DS_Store
+.DS_Store
diff --git a/README.md b/README.md
@@ -17,6 +17,41 @@ limitations under the License.
 
 ![logo](https://phoenix.apache.org/images/phoenix-logo-small.png)
 
-<b>[Apache Phoenix](http://phoenix.apache.org/)</b> enables OLTP and operational analytics in Hadoop for low latency applications. Visit the Apache Phoenix website <b>[here](http://phoenix.apache.org/)</b>. This is the repo for the Query Server. 
+<b>[Apache Phoenix](http://phoenix.apache.org/)</b> enables OLTP and operational analytics in Hadoop for low latency applications. Visit the Apache Phoenix website <b>[here](http://phoenix.apache.org/)</b>. This is the repo for the Phoenix Query Server (PQS).
 
-Copyright ©2019 [Apache Software Foundation](http://www.apache.org/). All Rights Reserved. 
+Copyright ©2020 [Apache Software Foundation](http://www.apache.org/). All Rights Reserved.
+
+## Introduction
+
+The Phoenix Query Server is an JDBC over HTTP abstraction. The Phoenix Query Server proxies the standard
+Phoenix JDBC driver and provides a backwards-compatible wire protocol to invoke that JDBC driver. This is
+all done via the Apache Avatica project (sub-project of Apache Calcite).
+
+The reference client implementation for PQS is a "thin" JDBC driver which can communicate with PQS. There
+are drivers in other languages which exist in varying levels of maturity including Python, Golang, and .NET.
+
+## Building
+
+This repository will build a tarball which is capable of running the Phoenix Query Server.
+
+By default, this tarball does not contain a Phoenix client jar as it is meant to be agnostic
+of Phoenix version (one PQS release can be used against any Phoenix version). Today, PQS builds against
+the Phoenix 4.15.0-HBase-1.4 release.
+
+```
+$ mvn package
+```
+
+### Bundling a Phoenix Client
+
+To build a release of PQS which packages a specific version of Phoenix, enable the `package-phoenix-client` profile
+and specify properties to indicate a specific Phoenix version.
+
+By default, PQS will package the same version of Phoenix used for build/test. This version is controlled by the system
+property `phoenix.version` system property. Depending on the version of Phoenix, you may also be required to
+use the `phoenix.hbase.classifier` system property to identify the correct version of Phoenix built against
+the version of HBase of your choosing.
+
+```
+$ mvn package -Dpackage.phoenix.client -Dphoenix.version=5.1.0-SNAPSHOT -Dphoenix.hbase.classifier=hbase-2.2
+```
diff --git a/assembly/pom.xml b/assembly/pom.xml
@@ -52,7 +52,6 @@
     </dependencies>
 
     <build>
-        <directory>target</directory>
         <plugins>
             <plugin>
                 <artifactId>maven-assembly-plugin</artifactId>
@@ -65,7 +64,7 @@
                         </goals>
                         <configuration>
                             <descriptors>
-                                <descriptor>cluster.xml</descriptor>
+                                <descriptor>src/assembly/cluster.xml</descriptor>
                             </descriptors>
                             <finalName>${project.parent.artifactId}-${project.version}</finalName>
                             <tarLongFileMode>posix</tarLongFileMode>
@@ -74,7 +73,43 @@
                     </execution>
                 </executions>
             </plugin>
+            <plugin>
+              <artifactId>maven-dependency-plugin</artifactId>
+              <executions>
+                <!-- Create a mini Maven repository so PQS can serve these jars like a Maven repo -->
+                <execution>
+                  <id>prepare-client-repo</id>
+                    <!-- Make sure we build this before making the assembly -->
+                  <phase>prepare-package</phase>
+                  <goals>
+                    <goal>copy-dependencies</goal>
+                  </goals>
+                  <configuration>
+                    <includeArtifactIds>phoenix-client,queryserver-client</includeArtifactIds>
+                    <outputDirectory>${project.build.directory}/maven-repo</outputDirectory>
+                    <overWriteIfNewer>true</overWriteIfNewer>
+                    <useRepositoryLayout>true</useRepositoryLayout>
+                  </configuration>
+                 </execution>
+              </executions>
+            </plugin>
         </plugins>
     </build>
-
+    <profiles>
+        <profile>
+            <id>package-phoenix-client</id>
+            <activation>
+              <property>
+                <name>package.phoenix.client</name>
+              </property>
+            </activation>
+            <dependencies>
+                <dependency>
+                    <groupId>org.apache.phoenix</groupId>
+                    <artifactId>phoenix-client</artifactId>
+                    <classifier>${phoenix.hbase.classifier}</classifier>
+                </dependency>
+            </dependencies>
+        </profile>
+    </profiles>
 </project>
diff --git a/assembly/cluster.xml → assembly/src/assembly/cluster.xml b/assembly/cluster.xml → assembly/src/assembly/cluster.xml
@@ -33,22 +33,21 @@
             <directory>${project.basedir}/../queryserver/target</directory>
             <outputDirectory>${project.parent.artifactId}-${project.parent.version}/queryserver/target</outputDirectory>
             <includes>
-                <include>*.jar</include>
+                <include>phoenix*.jar</include>
+                <include>queryserver*.jar</include>
             </includes>
         </fileSet>
         <fileSet>
             <directory>${project.basedir}/../queryserver-client/target</directory>
             <outputDirectory>${project.parent.artifactId}-${project.parent.version}/queryserver-client/target</outputDirectory>
             <includes>
-                <include>*.jar</include>
+                <include>phoenix*.jar</include>
+                <include>queryserver*.jar</include>
             </includes>
         </fileSet>
         <fileSet>
-            <directory>${project.basedir}/../phoenix-client/target</directory>
-            <outputDirectory>${project.parent.artifactId}-${project.parent.version}/phoenix-client/target</outputDirectory>
-            <includes>
-                <include>*.jar</include>
-            </includes>
+          <directory>${project.build.directory}/maven-repo</directory>
+          <outputDirectory>${project.parent.artifactId}-${project.parent.version}/maven</outputDirectory>
         </fileSet>
     </fileSets>
     <dependencySets>
@@ -59,5 +58,31 @@
           <include>sqlline:sqlline:jar:jar-with-dependencies</include>
         </includes>
       </dependencySet>
+      <dependencySet>
+        <unpack>false</unpack>
+        <outputDirectory>${project.parent.artifactId}-${project.parent.version}/</outputDirectory>
+        <includes>
+          <include>org.apache.phoenix:phoenix-client</include>
+        </includes>
+        <!-- Unwind the name from phoenix-client-$version to phonix-$version-client -->
+        <outputFileNameMapping>phoenix-${artifact.version}${dashClassifier}-client.${artifact.extension}</outputFileNameMapping>
+      </dependencySet>
+      <dependencySet>
+        <unpack>false</unpack>
+        <outputDirectory>${project.parent.artifactId}-${project.parent.version}/queryserver/target</outputDirectory>
+        <includes>
+          <include>org.apache.phoenix:queryserver</include>
+        </includes>
+        <outputFileNameMapping>phoenix-${project.parent.version}-queryserver.${artifact.extension}</outputFileNameMapping>
+      </dependencySet>
+      <!-- Unwind the name from queryserver-client-$version to phoenix-$version-thin-client -->
+      <dependencySet>
+        <unpack>false</unpack>
+        <outputDirectory>${project.parent.artifactId}-${project.parent.version}/queryserver-client/target</outputDirectory>
+        <includes>
+          <include>org.apache.phoenix:queryserver-client</include>
+        </includes>
+        <outputFileNameMapping>phoenix-${project.parent.version}-thin-client.${artifact.extension}</outputFileNameMapping>
+      </dependencySet>
     </dependencySets>
 </assembly>
diff --git a/load-balancer/pom.xml b/load-balancer/pom.xml
@@ -116,10 +116,6 @@
       <groupId>org.apache.curator</groupId>
       <artifactId>curator-framework</artifactId>
     </dependency>
-    <dependency>
-      <groupId>org.apache.calcite.avatica</groupId>
-      <artifactId>avatica</artifactId>
-    </dependency>
     <dependency>
       <groupId>org.slf4j</groupId>
       <artifactId>slf4j-api</artifactId>

diff --git a/pom.xml b/pom.xml
@@ -51,6 +51,7 @@
 
     <modules>
         <module>queryserver</module>
+        <module>queryserver-it</module>
         <module>queryserver-client</module>
         <module>load-balancer</module>
         <module>assembly</module>
@@ -67,7 +68,7 @@
         <!-- General Properties -->
         <top.dir>${project.basedir}</top.dir>
 
-        <phoenix.version>4.14.2-HBase-1.4</phoenix.version>
+        <phoenix.version>4.15.0-HBase-1.4</phoenix.version>
 
         <!-- Hadoop/Hbase Versions -->
         <hbase.version>1.4.10</hbase.version>
@@ -372,6 +373,12 @@
                     </exclusion>
                 </exclusions>
             </dependency>
+            <dependency>
+                <groupId>org.apache.hbase</groupId>
+                <artifactId>hbase-server</artifactId>
+                <version>${hbase.version}</version>
+                <scope>test</scope>
+            </dependency>
 
             <!-- Hadoop Dependencies -->
             <dependency>
@@ -422,6 +429,11 @@
                   </exclusion>
                 </exclusions>
             </dependency>
+            <dependency>
+              <groupId>org.eclipse.jetty</groupId>
+              <artifactId>jetty-util</artifactId>
+              <version>${jetty.version}</version>
+            </dependency>
             <dependency>
                 <groupId>org.apache.curator</groupId>
                 <artifactId>curator-client</artifactId>
@@ -480,6 +492,14 @@
                 <version>0.8.1</version>
             </dependency>
 
+            <!-- Phoenix test dependencies -->
+            <dependency>
+                <groupId>org.apache.phoenix</groupId>
+                <artifactId>queryserver</artifactId>
+                <version>${project.version}</version>
+                <classifier>tests</classifier>
+            </dependency>
+
             <!-- Hbase test dependencies -->
             <dependency>
                 <groupId>org.apache.hbase</groupId>
@@ -494,12 +514,6 @@
                     </exclusion>
                 </exclusions>
             </dependency>
-            <dependency>
-                <groupId>org.apache.hbase</groupId>
-                <artifactId>hbase-server</artifactId>
-                <version>${hbase.version}</version>
-                <scope>test</scope>
-            </dependency>
             <dependency>
                 <groupId>org.apache.hbase</groupId>
                 <artifactId>hbase-server</artifactId>
@@ -531,17 +545,10 @@
             </dependency>
 
             <!-- Other test dependencies -->
-            <dependency>
-              <groupId>org.eclipse.jetty</groupId>
-              <artifactId>jetty-util</artifactId>
-              <version>${jetty.version}</version>
-              <scope>test</scope>
-            </dependency>
             <dependency>
               <groupId>org.eclipse.jetty</groupId>
               <artifactId>jetty-security</artifactId>
               <version>${jetty.version}</version>
-              <scope>test</scope>
             </dependency>
             <dependency>
               <groupId>org.eclipse.jetty</groupId>
@@ -587,5 +594,28 @@
             </dependency>
         </dependencies>
     </dependencyManagement>
-
+    <profiles>
+        <profile>
+            <id>package-phoenix-client</id>
+            <activation>
+              <property>
+                <name>package.phoenix.client</name>
+              </property>
+            </activation>
+            <properties>
+                <!-- Not necessary for Phoenix <4.16.0 -->
+                <phoenix.hbase.classifier></phoenix.hbase.classifier>
+            </properties>
+            <dependencyManagement>
+                <dependencies>
+                    <dependency>
+                        <groupId>org.apache.phoenix</groupId>
+                        <artifactId>phoenix-client</artifactId>
+                        <version>${phoenix.version}</version>
+                        <classifier>${phoenix.hbase.classifier}</classifier>
+                    </dependency>
+                </dependencies>
+            </dependencyManagement>
+        </profile>
+    </profiles>
 </project>
diff --git a/queryserver-client/pom.xml b/queryserver-client/pom.xml
@@ -74,8 +74,7 @@
               <goal>shade</goal>
             </goals>
             <configuration>
-              <finalName>phoenix-${project.version}-thin-client</finalName>
-
+              <shadedArtifactAttached>false</shadedArtifactAttached>
               <transformers>
                 <transformer
                     implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">