[SPARK-7989][Core][Tests] Fix flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite #6546

zsxwing · 2015-06-01T03:04:55Z

The flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite will fail if there are not enough executors up before running the jobs.

This PR adds JobProgressListener.waitUntilExecutorsUp. The tests for the cluster mode can use it to wait until the expected executors are up.

…lusterSuite

SparkQA · 2015-06-01T04:53:56Z

Test build #33872 has finished for PR 6546 at commit 3b69840.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2015-06-01T09:29:07Z

@zsxwing are these things you identify actually causing test failures for you or in Jenkins or is this theoretical?

zsxwing · 2015-06-01T09:37:08Z

@srowen These two failures happened in my PR #6457. I fixed them in #6457 to pass Jenkins. But I think they are not a part of #6457 and it's better to fix them in a separate PR.

SparkQA · 2015-06-01T12:14:46Z

Test build #33885 timed out for PR 6546 at commit 5560e09 after a configured wait of 150m.

zsxwing · 2015-06-01T12:25:28Z

retest this please

SparkQA · 2015-06-01T14:16:48Z

Test build #33888 has finished for PR 6546 at commit 5560e09.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

squito · 2015-06-01T18:06:13Z

if this is just for testing, how about we put it into a SparkListener in the tests instead, and avoid changing JobProgressListener? You could put it into a common listener trait that could be used for both tests. The listener would just count how many times onExecutorAdded / onExecutorRemoved were called and would have a similar waitUntilExecutorsUp method.

zsxwing · 2015-06-01T23:44:56Z

if this is just for testing, how about we put it into a SparkListener in the tests instead, and avoid changing JobProgressListener? You could put it into a common listener trait that could be used for both tests. The listener would just count how many times onExecutorAdded / onExecutorRemoved were called and would have a similar waitUntilExecutorsUp method.

The problem of a new SparkListener is "adding an Executor" may happen before "adding the new SparkListener to SparkContext". So the new SparkListener may miss some messages before adding itself to SparkContext. JobProgressListener is added as early as possible so that it won't miss any message.

squito · 2015-06-02T00:39:53Z

The problem of a new SparkListener is "adding an Executor" may happen before "adding the new
SparkListener to SparkContext". So the new SparkListener may miss some messages before adding
itself to SparkContext. JobProgressListener is added as early as possible so that it won't miss any
message.

hmm, good point. You can make sure listeners are added immediately with spark.extraListeners, but that is admittedly a pain since you don't get a handle on our listener, so you need some to communicate via some global variable. I guess I don't feel strongly either way then.

andrewor14 · 2015-06-03T21:27:57Z

retest this please

andrewor14 · 2015-06-03T21:30:18Z

core/src/test/scala/org/apache/spark/ExternalShuffleServiceSuite.scala

@@ -55,6 +55,14 @@ class ExternalShuffleServiceSuite extends ShuffleSuite with BeforeAndAfterAll {
    sc.env.blockManager.externalShuffleServiceEnabled should equal(true)
    sc.env.blockManager.shuffleClient.getClass should equal(classOf[ExternalShuffleClient])

+    // In a slow machine, one slave may register hundreds of milliseconds ahead of the other one.
+    // If we don't wait for all salves, it's possible that only one executor runs all jobs. Then


andrewor14 · 2015-06-03T21:32:02Z

@zsxwing IIUC this only adds some wait logic for the two test suites, but the intent is that we don't change the logic in BroadcastSuite right?

andrewor14 · 2015-06-03T22:03:58Z

master 1.4

…Suite and SparkListenerWithClusterSuite The flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite will fail if there are not enough executors up before running the jobs. This PR adds `JobProgressListener.waitUntilExecutorsUp`. The tests for the cluster mode can use it to wait until the expected executors are up. Author: zsxwing <[email protected]> Closes #6546 from zsxwing/SPARK-7989 and squashes the following commits: 5560e09 [zsxwing] Fix a typo 3b69840 [zsxwing] Fix flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite (cherry picked from commit f271347) Signed-off-by: Andrew Or <[email protected]> Conflicts: core/src/test/scala/org/apache/spark/broadcast/BroadcastSuite.scala core/src/test/scala/org/apache/spark/scheduler/SparkListenerWithClusterSuite.scala

This is caused by 7e46ea0.

SparkQA · 2015-06-04T00:03:04Z

Test build #34116 timed out for PR 6546 at commit 5560e09 after a configured wait of 150m.

zsxwing · 2015-06-04T03:09:42Z

@zsxwing IIUC this only adds some wait logic for the two test suites, but the intent is that we don't change the logic in BroadcastSuite right?

Right. BroadcastSuite could also use this new method, so I updated it to simplify codes. And thanks for fixing the typos.

…Suite and SparkListenerWithClusterSuite The flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite will fail if there are not enough executors up before running the jobs. This PR adds `JobProgressListener.waitUntilExecutorsUp`. The tests for the cluster mode can use it to wait until the expected executors are up. Author: zsxwing <[email protected]> Closes apache#6546 from zsxwing/SPARK-7989 and squashes the following commits: 5560e09 [zsxwing] Fix a typo 3b69840 [zsxwing] Fix flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite

Fix flaky tests in ExternalShuffleServiceSuite and SparkListenerWithC…

3b69840

…lusterSuite

Fix a typo

5560e09

andrewor14 reviewed Jun 3, 2015
View reviewed changes

asfgit closed this in f271347 Jun 3, 2015

asfgit pushed a commit that referenced this pull request Jun 3, 2015

[HOTFIX] Unbreak build from backporting #6546

d0be950

This is caused by 7e46ea0.

asfgit pushed a commit that referenced this pull request Jun 3, 2015

[HOTFIX] [TYPO] Fix typo in #6546

0bc9a3e

asfgit pushed a commit that referenced this pull request Jun 3, 2015

[HOTFIX] [TYPO] Fix typo in #6546

bfbdab1

zsxwing deleted the SPARK-7989 branch June 4, 2015 03:09

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015

[HOTFIX] [TYPO] Fix typo in apache#6546

8b4636c

nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015

[HOTFIX] [TYPO] Fix typo in apache#6546

464e3f1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7989][Core][Tests] Fix flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite #6546

[SPARK-7989][Core][Tests] Fix flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite #6546

zsxwing commented Jun 1, 2015

SparkQA commented Jun 1, 2015

srowen commented Jun 1, 2015

zsxwing commented Jun 1, 2015

SparkQA commented Jun 1, 2015

zsxwing commented Jun 1, 2015

SparkQA commented Jun 1, 2015

squito commented Jun 1, 2015

zsxwing commented Jun 1, 2015

squito commented Jun 2, 2015

andrewor14 commented Jun 3, 2015

andrewor14 Jun 3, 2015

andrewor14 commented Jun 3, 2015

andrewor14 commented Jun 3, 2015

SparkQA commented Jun 4, 2015

zsxwing commented Jun 4, 2015

[SPARK-7989][Core][Tests] Fix flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite #6546

[SPARK-7989][Core][Tests] Fix flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite #6546

Conversation

zsxwing commented Jun 1, 2015

SparkQA commented Jun 1, 2015

srowen commented Jun 1, 2015

zsxwing commented Jun 1, 2015

SparkQA commented Jun 1, 2015

zsxwing commented Jun 1, 2015

SparkQA commented Jun 1, 2015

squito commented Jun 1, 2015

zsxwing commented Jun 1, 2015

squito commented Jun 2, 2015

andrewor14 commented Jun 3, 2015

andrewor14 Jun 3, 2015

Choose a reason for hiding this comment

andrewor14 commented Jun 3, 2015

andrewor14 commented Jun 3, 2015

SparkQA commented Jun 4, 2015

zsxwing commented Jun 4, 2015