[SPARK-14435][BUILD] Shade Kryo in our custom Hive 1.2.1 fork

[JoshRosen]

This patch updates our custom Hive 1.2.1 fork in order to shade Kryo in Hive. This is a blocker for upgrading Spark to use Kryo 3 (see #12076).

The source for this new fork of Hive can be found at https://github.com/JoshRosen/hive/tree/release-1.2.1-spark2

Here's the complete diff from the official Hive 1.2.1 release: apache/hive@release-1.2.1...JoshRosen:release-1.2.1-spark2

Here's the diff from the sources that @pwendell used to publish the current 1.2.1.spark release of Hive: pwendell/hive@release-1.2.1-spark...JoshRosen:release-1.2.1-spark2. This diff looks large because his branch used a shell script to rewrite the groupId, whereas I had to commit the groupId changes in order to prevent the find-and-replace from affecting the package names in our relocated Kryo classes: pwendell/hive@release-1.2.1-spark...JoshRosen:release-1.2.1-spark2#diff-6ada9aaec70e069df8f2c34c5519dd1e

Using these changes, I was able to publish a local version of Hive and verify that this change fixes the test failures which are blocking #12076. Note that this PR will not compile until we complete the review of the Hive POM changes and stage and publish a release.

/cc @vanzin, @steveloughran, and @pwendell for review.

[JoshRosen]

@vanzin, in light of #11941 do you think that we should change the namespace of our shaded protobuf to omit the hyphens?

[SparkQA]

Test build #55149 has finished for PR 12215 at commit 8f74c74.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Yeah, if we're rebuilding dependencies already, might as well avoid issues with the sort-of-illegal package names...

[JoshRosen]

Alright, updated to keep the relocated classes under the org.apache.hive namespace (like was previously done for Kryo):

diff --git a/ql/pom.xml b/ql/pom.xml
index 16f3d05..09a59bd 100644
--- a/ql/pom.xml
+++ b/ql/pom.xml
@@ -739,7 +739,7 @@
                 </relocation>
                 <relocation>
                   <pattern>com.google.protobuf</pattern>
-                  <shadedPattern>org.spark-project.hive.shaded.com.google.protobuf</shadedPattern>
+                  <shadedPattern>org.apache.hive.com.google.protobuf</shadedPattern>
                 </relocation>
               </relocations>
             </configuration>

[JoshRosen]

I have packaged and staged a new release of our custom Hive (I was careful to compile using JDK 7). Provided that tests pass here, I'm going to publish to Maven Central, wait for the new version to propagate, then loop back tomorrow and will commit this to unblock Kryo 3.

[JoshRosen]

There are two test failures, but they're both related to tests of the shading: one failed due to my removal of the hyphenated package namespace in relocated protobuf, and the other failed because it was explicitly checking for unshaded Kryo.

[JoshRosen]

Okay, so given that this passes all of the relevant Hive tests, I'm going to promote my 1.2.1-spark2 release to Maven Central, then will update this patch to prepare for final merge-readiness.

[SparkQA]

Test build #55269 has finished for PR 12215 at commit cdf266b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

org.spark-project.hive:hive-exec:1.2.1.spark2 is now on Maven Central, so this PR is no longer WIP.

[SparkQA]

Test build #55305 has finished for PR 12215 at commit 30eb58d.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #55294 has finished for PR 12215 at commit e2237bd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #55311 has finished for PR 12215 at commit 30eb58d.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins retest this please

[SparkQA]

Test build #55332 has finished for PR 12215 at commit 30eb58d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

LGTM

[JoshRosen]

Alright, I'm going to merge this to master and will resume work on the other patch for the Kryo 3 migration.