[SPARK-33212][FOLLOWUP] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile #31642

sunchao · 2021-02-25T06:50:44Z

What changes were proposed in this pull request?

This adds hadoop-yarn-server-web-proxy as dependency for Yarn and Hadoop 3.x profile (it is already a dependency for 2.x). Also excludes some dependencies from the module which are already covered by other Hadoop jars used by Spark.

Why are the changes needed?

The class org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter is used by ApplicationMaster:

  private def addAmIpFilter(driver: Option[RpcEndpointRef], proxyBase: String) = {
    val amFilter = "org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter"
    val params = client.getAmIpFilterParams(yarnConf, proxyBase)
    driver match {
      case Some(d) =>
        d.send(AddWebUIFilter(amFilter, params, proxyBase))
   ...

and will be loaded at runtime. Therefore, without the above jar Spark Yarn app will fail with ClassNotFoundError.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing unit tests. Also tested manually and it worked with the fix, while was failing previously.

SparkQA · 2021-02-25T08:12:31Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40039/

SparkQA · 2021-02-25T08:16:53Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40039/

SparkQA · 2021-02-25T09:41:40Z

Test build #135459 has finished for PR 31642 at commit 17a63bd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

xkrogen · 2021-02-25T16:36:34Z

This seems like a decent workaround, but is it the right long-term fix? It seems like depending on the non-shaded JAR breaks the benefits of leveraging only the shaded hadoop-client-api and hadoop-client-runtime JARs.

Should AmIpFilter be a public class, i.e., is it something AMs are expected to leverage? If yes -- can we get it moved to hadoop-client-runtime? If no -- should we stop using it here and re-implement some of the required logic ourselves?

sunchao · 2021-02-25T22:01:26Z

This is a small jar though and we've striped all its transitive dependencies. Agree that in the long term we should either: 1) consider to re-implement the logic in Spark which allows us to get away from server-side dependency in Hadoop. cc @tgravescs for more insights on this. 2) move the class to hadoop-client-runtime as you suggested.

I need to take a look at the test failures - seems WebAppProxyServlet was pulled from hadoop-yarn-server-web-proxy jar instead of hadoop-client-minicluster.

dongjoon-hyun · 2021-02-25T23:53:07Z

Retest this please

dongjoon-hyun · 2021-02-25T23:53:42Z

Thank you for the follow-up, @sunchao !

…p 3.x profile

sunchao · 2021-02-26T01:14:01Z

assembly/pom.xml

@@ -136,6 +136,10 @@
          <artifactId>spark-yarn_${scala.binary.version}</artifactId>
          <version>${project.version}</version>
        </dependency>
+        <dependency>
+          <groupId>org.apache.hadoop</groupId>


This artifact is only needed at runtime so included here in distribution. Ideally we should declare it as a runtime dependency. However, the WebAppProxyServlet class in the jar conflicts with the same one from hadoop-cluster-client-minicluster during tests.

SparkQA · 2021-02-26T01:38:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40063/

SparkQA · 2021-02-26T02:07:44Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40063/

SparkQA · 2021-02-26T02:27:55Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40065/

SparkQA · 2021-02-26T02:54:39Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40065/

dongjoon-hyun · 2021-02-26T03:17:18Z

@sunchao . The PR title looks misleading. Which hadoop profile are you testing now?

dongjoon-hyun · 2021-02-26T03:19:29Z

I checked the Jenkins log. It's testing Hadoop 3.2 correctly. I'll fix the PR title.

[info] Building Spark using Maven with these arguments:  -Phadoop-3.2 -Phive-2.3 -Pyarn -Phadoop-cloud -Pkinesis-asl -Pkubernetes -Phive-thriftserver -Phive -Pspark-ganglia-lgpl -Pmesos clean package -DskipTests

dongjoon-hyun

+1, LGTM (Pending Jenkins)

sunchao · 2021-02-26T03:26:35Z

Sorry I wanted to test 2.7 and 3.2 at the same time. Is that possible?

SparkQA · 2021-02-26T04:06:52Z

Test build #135483 has finished for PR 31642 at commit 17a63bd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-02-26T04:36:25Z

It's not possible for the single title because one Jenkins run only will pick-up a single profile.

What you need is to run two Jenkins runs. So, you need to trigger twice like the following.

Set the title with test-hadoop3.2 and trigger.
Wait 5 or 10 minutes until Jenkins posts a comment. Then, check the log.
Then, rename the title with test-hadoop3.2 and trigger.
Wait another 5 or 10 minutes until Jenkins posts a comment. Then, check the log.

Then, you will have effectively concurrent two runs (for both profiles)

SparkQA · 2021-02-26T05:14:16Z

Test build #135485 has finished for PR 31642 at commit e2d32aa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sunchao · 2021-02-26T05:53:52Z

Thanks @dongjoon-hyun . Let me trigger another run for Hadoop 2.7 just to be sure.

Jenkins CI says there're 2 failures in SQL module but I'm guessing they're not related (this PR doesn't touch anything in SQL), although I can't seem to find which two tests are these.

SparkQA · 2021-02-26T18:28:04Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40100/

SparkQA · 2021-02-26T18:57:34Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40100/

SparkQA · 2021-02-26T21:26:47Z

Test build #135519 has finished for PR 31642 at commit 1de8590.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-02-27T04:27:47Z

Test build #135525 has finished for PR 31642 at commit e2d32aa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-02-27T06:30:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40113/

SparkQA · 2021-02-27T06:59:06Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40113/

SparkQA · 2021-02-27T08:11:27Z

Test build #135532 has finished for PR 31642 at commit c31e06d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-02-27T18:35:54Z

Could you confirm which links are the two successful runs (Hadoop 3.2/2.7)?

sunchao · 2021-02-28T05:47:09Z

@dongjoon-hyun Github CI all succeeded (except the flaky tests).

For the Jenkins CI, here are the results:

3.2
2.7

The tests all passed except two in SQL:

Caused by: java.lang.IllegalArgumentException: Unknown key pii
	at org.apache.orc.impl.HadoopShimsPre2_3$NullKeyProvider.getCurrentKeyVersion(HadoopShimsPre2_3.java:71)
	at org.apache.orc.impl.WriterImpl.getKey(WriterImpl.java:871)
	at org.apache.orc.impl.WriterImpl.visitTypeTree(WriterImpl.java:902)
	at org.apache.orc.impl.WriterImpl.visitTypeTree(WriterImpl.java:915)
	at org.apache.orc.impl.WriterImpl.setupEncryption(WriterImpl.java:938)
	at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:162)
	at org.apache.orc.OrcFile.createWriter(OrcFile.java:1004)
	at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:47)
	at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:120)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:126)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:111)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:270)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:210)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
- Write and read an encrypted file *** FAILED ***
  org.apache.spark.SparkException: Job aborted.
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:232)
  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131)
  at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
  at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)

which doesn't look like related as this PR only touches dependencies for YARN.

Also the above succeeded in Jenkins CI if I remove [test-maven].

dongjoon-hyun · 2021-03-01T00:34:42Z

Thank you for the summary. For those test cases, I'll take a look in this week, @sunchao .

dongjoon-hyun

+1, LGTM. The two failures are not related to this PR.
I'll merge this.

tgravescs · 2021-03-01T16:38:48Z

yes AmIpFilter is expected to be used by ApplicationMasters so ideally Hadoop would move it to somewhere easily accessible.

….x profile This adds `hadoop-yarn-server-web-proxy` as dependency for Yarn and Hadoop 3.x profile (it is already a dependency for 2.x). Also excludes some dependencies from the module which are already covered by other Hadoop jars used by Spark. The class `org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter` is used by `ApplicationMaster`: ```scala private def addAmIpFilter(driver: Option[RpcEndpointRef], proxyBase: String) = { val amFilter = "org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter" val params = client.getAmIpFilterParams(yarnConf, proxyBase) driver match { case Some(d) => d.send(AddWebUIFilter(amFilter, params, proxyBase)) ... ``` and will be loaded at runtime. Therefore, without the above jar Spark Yarn app will fail with `ClassNotFoundError`. No. Existing unit tests. Also tested manually and it worked with the fix, while was failing previously. Closes apache#31642 from sunchao/SPARK-33212-followup-2. Authored-by: Chao Sun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…ementation ### What changes were proposed in this pull request? This PR replaces AmIpFilter with a forked implementation, and removes the dependency `hadoop-yarn-server-web-proxy` ### Why are the changes needed? SPARK-47118 upgraded Spark built-in Jetty from 10 to 11, and migrated from `javax.servlet` to `jakarta.servlet`, which breaks the Spark on YARN. ``` Caused by: java.lang.IllegalStateException: class org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter at org.sparkproject.jetty.servlet.FilterHolder.doStart(FilterHolder.java:99) at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) at org.sparkproject.jetty.servlet.ServletHandler.lambda$initialize$2(ServletHandler.java:724) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:749) ... 38 more ``` During the investigation, I found a comment here #31642 (comment) > Agree that in the long term we should either: 1) consider to re-implement the logic in Spark which allows us to get away from server-side dependency in Hadoop ... This should be a simple and clean way to address the exact issue, then we don't need to wait for Hadoop `jakarta.servlet` migration, and it also strips a Hadoop dependency. ### Does this PR introduce _any_ user-facing change? No, this recovers the bootstrap of the Spark application on YARN mode, keeping the same behavior with Spark 3.5 and earlier versions. ### How was this patch tested? UTs are added. (refer to `org.apache.hadoop.yarn.server.webproxy.amfilter.TestAmFilter`) I tested it in a YARN cluster. Spark successfully started. ``` roothadoop-master1:/opt/spark-SPARK-48238# JAVA_HOME=/opt/openjdk-17 bin/spark-sql --conf spark.yarn.appMasterEnv.JAVA_HOME=/opt/openjdk-17 --conf spark.executorEnv.JAVA_HOME=/opt/openjdk-17 WARNING: Using incubator modules: jdk.incubator.vector Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 2024-05-18 04:11:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-05-18 04:11:44 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive} is set, falling back to uploading libraries under SPARK_HOME. Spark Web UI available at http://hadoop-master1.orb.local:4040 Spark master: yarn, Application Id: application_1716005503866_0001 spark-sql (default)> select version(); 4.0.0 4ddc230 Time taken: 1.707 seconds, Fetched 1 row(s) spark-sql (default)> ``` When access `http://hadoop-master1.orb.local:4040`, it redirects to `http://hadoop-master1.orb.local:8088/proxy/redirect/application_1716005503866_0001/`, and the UI looks correct. <img width="1474" alt="image" src="https://github.com/apache/spark/assets/26535726/8500fc83-48c5-4603-8d05-37855f0308ae"> ### Was this patch authored or co-authored using generative AI tooling? No Closes #46611 from pan3793/SPARK-48238. Authored-by: Cheng Pan <[email protected]> Signed-off-by: yangjie01 <[email protected]>

…ementation ### What changes were proposed in this pull request? This PR replaces AmIpFilter with a forked implementation, and removes the dependency `hadoop-yarn-server-web-proxy` ### Why are the changes needed? SPARK-47118 upgraded Spark built-in Jetty from 10 to 11, and migrated from `javax.servlet` to `jakarta.servlet`, which breaks the Spark on YARN. ``` Caused by: java.lang.IllegalStateException: class org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter at org.sparkproject.jetty.servlet.FilterHolder.doStart(FilterHolder.java:99) at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) at org.sparkproject.jetty.servlet.ServletHandler.lambda$initialize$2(ServletHandler.java:724) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:749) ... 38 more ``` During the investigation, I found a comment here apache/spark#31642 (comment) > Agree that in the long term we should either: 1) consider to re-implement the logic in Spark which allows us to get away from server-side dependency in Hadoop ... This should be a simple and clean way to address the exact issue, then we don't need to wait for Hadoop `jakarta.servlet` migration, and it also strips a Hadoop dependency. ### Does this PR introduce _any_ user-facing change? No, this recovers the bootstrap of the Spark application on YARN mode, keeping the same behavior with Spark 3.5 and earlier versions. ### How was this patch tested? UTs are added. (refer to `org.apache.hadoop.yarn.server.webproxy.amfilter.TestAmFilter`) I tested it in a YARN cluster. Spark successfully started. ``` roothadoop-master1:/opt/spark-SPARK-48238# JAVA_HOME=/opt/openjdk-17 bin/spark-sql --conf spark.yarn.appMasterEnv.JAVA_HOME=/opt/openjdk-17 --conf spark.executorEnv.JAVA_HOME=/opt/openjdk-17 WARNING: Using incubator modules: jdk.incubator.vector Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 2024-05-18 04:11:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-05-18 04:11:44 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive} is set, falling back to uploading libraries under SPARK_HOME. Spark Web UI available at http://hadoop-master1.orb.local:4040 Spark master: yarn, Application Id: application_1716005503866_0001 spark-sql (default)> select version(); 4.0.0 4ddc2303c7cbabee12a3de9f674aaacad3f5eb01 Time taken: 1.707 seconds, Fetched 1 row(s) spark-sql (default)> ``` When access `http://hadoop-master1.orb.local:4040`, it redirects to `http://hadoop-master1.orb.local:8088/proxy/redirect/application_1716005503866_0001/`, and the UI looks correct. <img width="1474" alt="image" src="https://github.com/apache/spark/assets/26535726/8500fc83-48c5-4603-8d05-37855f0308ae"> ### Was this patch authored or co-authored using generative AI tooling? No Closes #46611 from pan3793/SPARK-48238. Authored-by: Cheng Pan <[email protected]> Signed-off-by: yangjie01 <[email protected]>

github-actions bot added BUILD YARN labels Feb 25, 2021

sunchao force-pushed the SPARK-33212-followup-2 branch from 17a63bd to 422d5ca Compare February 26, 2021 01:09

[SPARK-33212] Include hadoop-yarn-server-web-proxy for Yarn and Hadoo…

e2d32aa

…p 3.x profile

sunchao force-pushed the SPARK-33212-followup-2 branch from 422d5ca to e2d32aa Compare February 26, 2021 01:10

sunchao commented Feb 26, 2021

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile~~ [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile Feb 26, 2021

dongjoon-hyun approved these changes Feb 26, 2021

View reviewed changes

sunchao changed the title ~~[SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile~~ [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile Feb 26, 2021

sunchao force-pushed the SPARK-33212-followup-2 branch from 1de8590 to e2d32aa Compare February 27, 2021 00:14

sunchao changed the title ~~[SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile~~ [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile Feb 27, 2021

Empty commit to trigger CI again

c31e06d

dongjoon-hyun changed the title ~~[SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile~~ [SPARK-33212][FOLLOWUP] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile Mar 1, 2021

dongjoon-hyun approved these changes Mar 1, 2021

View reviewed changes

dongjoon-hyun closed this in f494c5c Mar 1, 2021

pan3793 mentioned this pull request May 16, 2024

[SPARK-48238][BUILD][YARN] Replace YARN AmIpFilter with a forked implementation #46611

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-33212][FOLLOWUP] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile #31642

[SPARK-33212][FOLLOWUP] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile #31642

sunchao commented Feb 25, 2021

SparkQA commented Feb 25, 2021

SparkQA commented Feb 25, 2021

SparkQA commented Feb 25, 2021

xkrogen commented Feb 25, 2021

sunchao commented Feb 25, 2021

dongjoon-hyun commented Feb 25, 2021

dongjoon-hyun commented Feb 25, 2021

sunchao Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 26, 2021

dongjoon-hyun commented Feb 26, 2021

dongjoon-hyun commented Feb 26, 2021

dongjoon-hyun left a comment

sunchao commented Feb 26, 2021

SparkQA commented Feb 26, 2021

dongjoon-hyun commented Feb 26, 2021

SparkQA commented Feb 26, 2021

sunchao commented Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 27, 2021

SparkQA commented Feb 27, 2021

SparkQA commented Feb 27, 2021

SparkQA commented Feb 27, 2021

dongjoon-hyun commented Feb 27, 2021

sunchao commented Feb 28, 2021

dongjoon-hyun commented Mar 1, 2021

dongjoon-hyun left a comment

tgravescs commented Mar 1, 2021

[SPARK-33212][FOLLOWUP] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile #31642

[SPARK-33212][FOLLOWUP] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile #31642

Conversation

sunchao commented Feb 25, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Feb 25, 2021

SparkQA commented Feb 25, 2021

SparkQA commented Feb 25, 2021

xkrogen commented Feb 25, 2021

sunchao commented Feb 25, 2021

dongjoon-hyun commented Feb 25, 2021

dongjoon-hyun commented Feb 25, 2021

sunchao Feb 26, 2021

Choose a reason for hiding this comment

SparkQA commented Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 26, 2021

dongjoon-hyun commented Feb 26, 2021

dongjoon-hyun commented Feb 26, 2021

dongjoon-hyun left a comment

Choose a reason for hiding this comment

sunchao commented Feb 26, 2021

SparkQA commented Feb 26, 2021

dongjoon-hyun commented Feb 26, 2021

SparkQA commented Feb 26, 2021

sunchao commented Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 26, 2021

SparkQA commented Feb 27, 2021

SparkQA commented Feb 27, 2021

SparkQA commented Feb 27, 2021

SparkQA commented Feb 27, 2021

dongjoon-hyun commented Feb 27, 2021

sunchao commented Feb 28, 2021

dongjoon-hyun commented Mar 1, 2021

dongjoon-hyun left a comment

Choose a reason for hiding this comment

tgravescs commented Mar 1, 2021