[SPARK-31101][BUILD] Upgrade Janino to 3.0.16 #27932

HeartSaVioR · 2020-03-17T04:14:10Z

What changes were proposed in this pull request?

This PR(SPARK-31101) proposes to upgrade Janino to 3.0.16 which is released recently.

Merged pull request Issue #113: Grow the code for relocatables, and do fixup, and relocate janino-compiler/janino#114 "Grow the code for relocatables, and do fixup, and relocate".

Please see the commit log.

https://github.com/janino-compiler/janino/commits/3.0.16

You can see the changelog from the link: http://janino-compiler.github.io/janino/changelog.html / though release note for Janino 3.0.16 is actually incorrect.

Why are the changes needed?

We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details.
Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing UTs.

…lish via Jitpack...

SparkQA · 2020-03-17T04:26:05Z

Test build #119906 has finished for PR 27932 at commit 1bf9e7d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-17T06:50:08Z

Test build #119908 has finished for PR 27932 at commit 78f8021.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-17T06:54:14Z

The failure is not from compilation from generated code, but no big deal on rerunning tests.

HeartSaVioR · 2020-03-17T06:54:21Z

retest this, please

SparkQA · 2020-03-17T09:56:49Z

Test build #119918 has finished for PR 27932 at commit 78f8021.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-17T23:33:34Z

retest this, please

SparkQA · 2020-03-18T02:19:58Z

Test build #119956 has finished for PR 27932 at commit 78f8021.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-18T07:19:35Z

retest this, please

SparkQA · 2020-03-18T09:33:29Z

Test build #119974 has finished for PR 27932 at commit 78f8021.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-18T10:25:28Z

retest this, please

SparkQA · 2020-03-18T13:08:27Z

Test build #119984 has finished for PR 27932 at commit 78f8021.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-19T03:14:57Z

Test build #120016 has finished for PR 27932 at commit ff25ac5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-19T03:43:30Z

org.apache.spark.scheduler.BarrierTaskContextSuite.successively sync with allGather and barrier
 Error Details
org.apache.spark.SparkException: Job aborted due to stage failure: Could not recover from a failed barrier ResultStage. Most recent failure reason: Stage failed because barrier task ResultTask(0, 3) finished unsuccessfully. com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input  at [Source: (String)""; line: 1, column: 0]  at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)  at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1442)  at com.fasterxml.jackson.databind.ObjectReader._initForReading(ObjectReader.java:360)  at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1704)  at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1261)  at org.json4s.jackson.JsonMethods.parse(JsonMethods.scala:25)  at org.json4s.jackson.JsonMethods.parse$(JsonMethods.scala:19)  at org.json4s.jackson.JsonMethods$.parse(JsonMethods.scala:55)  at org.apache.spark.BarrierTaskContext.allGather(BarrierTaskContext.scala:222)  at org.apache.spark.scheduler.BarrierTaskContextSuite.$anonfun$new$12(BarrierTaskContextSuite.scala:118)  at org.apache.spark.rdd.RDDBarrier.$anonfun$mapPartitions$2(RDDBarrier.scala:51)  at org.apache.spark.rdd.RDDBarrier.$anonfun$mapPartitions$2$adapted(RDDBarrier.scala:51)  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)  at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)  at org.apache.spark.scheduler.Task.run(Task.scala:127)  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:460)  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:463)  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)  at java.base/java.lang.Thread.run(Thread.java:834)

Doesn't seem to be related.

HeartSaVioR · 2020-03-19T03:43:55Z

retest this, please

SparkQA · 2020-03-19T06:37:31Z

Test build #120019 has finished for PR 27932 at commit ff25ac5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-19T06:43:53Z

org.apache.spark.ExecutorAllocationManagerSuite.add executors capped by num pending tasks 

Error Details

org.scalatest.exceptions.TestFailedException: 6 did not equal 5

org.apache.spark.scheduler.WorkerDecommissionSuite.verify a task with all workers decommissioned succeeds 

Error Details

java.util.concurrent.TimeoutException: Futures timed out after [2 seconds]

org.apache.spark.storage.BlockManagerReplicationSuite.block replication - mixed between 1x to 5x 

Error Details

org.scalatest.exceptions.TestFailedException: 3 did not equal 4 master did not have 4 locations for test_block-with-disk-memory-deserialized-4x-replicated

Another flaky ones.

SparkQA · 2020-03-19T10:10:38Z

Test build #120030 has finished for PR 27932 at commit a9f1f5d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM. Merged to master.
Thank you for your long investigating and patience, @HeartSaVioR .

I added the following into the PR description.

https://github.com/janino-compiler/janino/commits/3.0.16

HeartSaVioR · 2020-03-24T02:19:00Z

Thanks for sorting it out. Could we port back to 3.0 & 2.4 as well, given the origin bug was from Spark 2.3? I'll submit PRs for these versions. Thanks again!

### What changes were proposed in this pull request? This PR(SPARK-31101) proposes to upgrade Janino to 3.0.16 which is released recently. * Merged pull request janino-compiler/janino#114 "Grow the code for relocatables, and do fixup, and relocate". Please see the commit log. - https://github.com/janino-compiler/janino/commits/3.0.16 You can see the changelog from the link: http://janino-compiler.github.io/janino/changelog.html / though release note for Janino 3.0.16 is actually incorrect. ### Why are the changes needed? We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details. Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing UTs. Closes apache#27932 from HeartSaVioR/SPARK-31101-janino-3.0.16. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

This PR(SPARK-31101) proposes to upgrade Janino to 3.0.16 which is released recently. * Merged pull request janino-compiler/janino#114 "Grow the code for relocatables, and do fixup, and relocate". Please see the commit log. - https://github.com/janino-compiler/janino/commits/3.0.16 You can see the changelog from the link: http://janino-compiler.github.io/janino/changelog.html / though release note for Janino 3.0.16 is actually incorrect. We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details. Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly. No. Existing UTs. Closes apache#27932 from HeartSaVioR/SPARK-31101-janino-3.0.16. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

HeartSaVioR · 2020-03-24T05:54:12Z

#27996 for Spark 3.0.0 / #27997 for Spark 2.4.x

### What changes were proposed in this pull request? This PR(SPARK-31101) proposes to upgrade Janino to 3.0.16 which is released recently. * Merged pull request janino-compiler/janino#114 "Grow the code for relocatables, and do fixup, and relocate". Please see the commit log. - https://github.com/janino-compiler/janino/commits/3.0.16 You can see the changelog from the link: http://janino-compiler.github.io/janino/changelog.html / though release note for Janino 3.0.16 is actually incorrect. ### Why are the changes needed? We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details. Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing UTs. Closes apache#27932 from HeartSaVioR/SPARK-31101-janino-3.0.16. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

HeartSaVioR added 2 commits March 17, 2020 13:06

[SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1

7f266d7

WIP! Bump up Janino to 3.0.15 + apache#113 + some modification to pub…

1bf9e7d

…lish via Jitpack...

Roll back the change only applicable in Janino 3.1.x

78f8021

HeartSaVioR mentioned this pull request Mar 17, 2020

[SPARK-31115][SQL] Detect known Janino bug janino-compiler/janino#113 and apply workaround automatically as a fail-back via avoid using switch statement in generated code #27872

Closed

HeartSaVioR changed the title ~~[WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16~~ [WIP][SPARK-31101][SQL][BUILD][test-java11] Upgrade Janino to 3.0.16 Mar 17, 2020

HeartSaVioR changed the title ~~[WIP][SPARK-31101][SQL][BUILD][test-java11] Upgrade Janino to 3.0.16~~ [WIP][SPARK-31101][SQL][BUILD][test-hadoop3.2][test-java11] Upgrade Janino to 3.0.16 Mar 18, 2020

HeartSaVioR changed the title ~~[WIP][SPARK-31101][SQL][BUILD][test-hadoop3.2][test-java11] Upgrade Janino to 3.0.16~~ [SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 Mar 19, 2020

Apply Janino 3.0.16, and remove all hack around using Jitpack

a9f1f5d

HeartSaVioR force-pushed the SPARK-31101-janino-3.0.16 branch from ff25ac5 to a9f1f5d Compare March 19, 2020 07:07

dongjoon-hyun added the BUILD label Mar 22, 2020

dongjoon-hyun changed the title ~~[SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16~~ [SPARK-31101][BUILD] Upgrade Janino to 3.0.16 Mar 22, 2020

dongjoon-hyun approved these changes Mar 22, 2020

View reviewed changes

dongjoon-hyun closed this in f55f6b5 Mar 22, 2020

HeartSaVioR deleted the SPARK-31101-janino-3.0.16 branch March 24, 2020 05:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-31101][BUILD] Upgrade Janino to 3.0.16 #27932

[SPARK-31101][BUILD] Upgrade Janino to 3.0.16 #27932

HeartSaVioR commented Mar 17, 2020 •

edited by dongjoon-hyun

Loading

SparkQA commented Mar 17, 2020

SparkQA commented Mar 17, 2020

HeartSaVioR commented Mar 17, 2020

HeartSaVioR commented Mar 17, 2020

SparkQA commented Mar 17, 2020

HeartSaVioR commented Mar 17, 2020

SparkQA commented Mar 18, 2020

HeartSaVioR commented Mar 18, 2020

SparkQA commented Mar 18, 2020

HeartSaVioR commented Mar 18, 2020

SparkQA commented Mar 18, 2020

SparkQA commented Mar 19, 2020

HeartSaVioR commented Mar 19, 2020

HeartSaVioR commented Mar 19, 2020

SparkQA commented Mar 19, 2020

HeartSaVioR commented Mar 19, 2020

SparkQA commented Mar 19, 2020

dongjoon-hyun left a comment •

edited

Loading

HeartSaVioR commented Mar 24, 2020

HeartSaVioR commented Mar 24, 2020

[SPARK-31101][BUILD] Upgrade Janino to 3.0.16 #27932

[SPARK-31101][BUILD] Upgrade Janino to 3.0.16 #27932

Conversation

HeartSaVioR commented Mar 17, 2020 • edited by dongjoon-hyun Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Mar 17, 2020

SparkQA commented Mar 17, 2020

HeartSaVioR commented Mar 17, 2020

HeartSaVioR commented Mar 17, 2020

SparkQA commented Mar 17, 2020

HeartSaVioR commented Mar 17, 2020

SparkQA commented Mar 18, 2020

HeartSaVioR commented Mar 18, 2020

SparkQA commented Mar 18, 2020

HeartSaVioR commented Mar 18, 2020

SparkQA commented Mar 18, 2020

SparkQA commented Mar 19, 2020

HeartSaVioR commented Mar 19, 2020

HeartSaVioR commented Mar 19, 2020

SparkQA commented Mar 19, 2020

HeartSaVioR commented Mar 19, 2020

SparkQA commented Mar 19, 2020

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

HeartSaVioR commented Mar 24, 2020

HeartSaVioR commented Mar 24, 2020

HeartSaVioR commented Mar 17, 2020 •

edited by dongjoon-hyun

Loading

dongjoon-hyun left a comment •

edited

Loading