[SPARK-30051][BUILD] Clean up hadoop-3.2 dependency

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to cut org.eclipse.jetty:jetty-webappand org.eclipse.jetty:jetty-xml transitive dependency from hadoop-common.

Why are the changes needed?

This will simplify our dependency management by the removal of unused dependencies.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the GitHub Action with all combinations and the Jenkins UT with (Hadoop-3.2).

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #114744 has finished for PR 26742 at commit b326f31.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #114745 has finished for PR 26742 at commit b326f31.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @srowen and @HyukjinKwon .
Could you review this PR?

[dongjoon-hyun]

The hadoop-3.2-specific exclusion starts from https://github.com/apache/spark/pull/26742/files#diff-600376dffeb79835ede4a0b285078036R1034 .

[srowen]

What's the logic here -- it's not actually needed at compile time because?

[dongjoon-hyun]

spark-core already embeds the required jetty libraries into spark-core_2.13_*.jar, and these files are not used even in our UTs. Ur, are we using these libraries?

[dongjoon-hyun]

If I misunderstand something, I'll close this PR~ Please let me know.

[srowen]

I may also be missing something, but jetty is still used throughout the code. This seems to remove it entirely from the distribution by marking it provided. Given that it's directly used it seems to need to be compile scope and be included. I do agree with a direct dependency on it if it's directly used.

[dongjoon-hyun]

Before this PR, I tested with make-distribution.sh and master/worker/historyserver/sparkthriftserver/shell (I didn't test the yarn suffle service though).

This seems to remove it entirely from the distribution by marking it provided.

1. Yes. This PR removes two files because we embedded needed jetty into spark-core library already. The following is the result of make-distribution.sh.

$ jar tvf spark-core_2.12-3.0.0-SNAPSHOT.jar | grep jetty | tail -n5
 11996 Mon Dec 02 22:25:20 PST 2019 org/sparkproject/jetty/servlets/PushCacheFilter.class
  9569 Mon Dec 02 22:25:20 PST 2019 org/sparkproject/jetty/servlets/PutFilter.class
  1302 Mon Dec 02 22:25:20 PST 2019 org/sparkproject/jetty/servlets/CGI$3.class
   108 Mon Apr 29 15:50:58 PDT 2019 META-INF/maven/org.eclipse.jetty/jetty-servlets/pom.properties
  2142 Mon Dec 02 22:25:20 PST 2019 org/sparkproject/jetty/servlets/CGI$EnvList.class

2. Jetty project consists of multi-modules like Apache Spark. jetty-webapp is something like graphx. jetty-webapp has WebAppContext which loads web.xml and builds web apps.

A WebAppContext is a derivation of ServletContextHandler that supports the standardized layout of a web application and configuration of session, security, listeners, filter, servlets, and JSP via a web.xml descriptor normally found in the /WEB-INF directory of a web application.

https://github.com/eclipse/jetty.project/tree/jetty-9.4.x/jetty-webapp

AFAIK, Apache Spark don't use web.xml or the standardized-layouted web applications style. I guess yarn shuffle-service also doesn't use this.

[srowen]

OK yeah I agree the webapp artifact isn't used directly. It isn't in the other deps files either. When you say embedded, you mean shaded? Yes it is.

I guess I'm confused why we don't see the artifact in the deps list anyway, even if shaded, or why we only see a few still in the Hadoop 2.x builds like jetty-util, but, maybe I just don't recall how it works. OK well it looks consistent at least.

[dongjoon-hyun]

Thank you for review. Merged to master.

[HyukjinKwon]

Looks fine to me too