Add tests for FileLogger, EventLoggingListener, and ReplayListenerBus #591
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
An event can be mutated by the DAGScheduler in between being procssed by one listener and being processed by another. This causes the ReplayListenerSuite to be flaky. This commit ensures that the event logged is the same as the original event received by the EventLoggingListener.
Conflicts: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala core/src/main/scala/org/apache/spark/util/FileLogger.scala
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
| @@ -64,7 +70,8 @@ private[spark] class EventLoggingListener( | |||
| def start() { | |||
| logInfo("Logging events to %s".format(logDir)) | |||
| if (shouldCompress) { | |||
| val codec = conf.get("spark.io.compression.codec", CompressionCodec.DEFAULT_COMPRESSION_CODEC) | |||
There was a problem hiding this comment.
I don't think this was over the line limit before (?) wouldn't it have failed our style checks?
There was a problem hiding this comment.
I renamed it to sparkConf, because there's also a hadoopConf now
There was a problem hiding this comment.
Ah I see - I misread this to only be a linebreak change
|
Unfortunately I only had time to do a cursory glance. I mostly just sanity checked the changes to the non-test code to make sure there were no mistakes. Looks good to me pending some small comments about the tests. |
|
Build triggered. |
|
Build started. |
Conflicts: core/src/main/scala/org/apache/spark/util/Utils.scala
|
Merged build triggered. |
|
Merged build started. |
|
Build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14597/ |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Thanks - I merged this. |
asfgit
pushed a commit
that referenced
this pull request
May 2, 2014
Modifications to Spark core are limited to exposing functionality to test files + minor style fixes. (728 / 769 lines are from tests) Author: Andrew Or <[email protected]> Closes #591 from andrewor14/event-log-tests and squashes the following commits: 2883837 [Andrew Or] Merge branch 'master' of github.com:apache/spark into event-log-tests c3afcea [Andrew Or] Compromise 2d5daf8 [Andrew Or] Use temp directory provided by the OS rather than /tmp 2b52151 [Andrew Or] Remove unnecessary file delete + add a comment 62010fd [Andrew Or] More cleanup (renaming variables, updating comments etc) ad2beff [Andrew Or] Clean up EventLoggingListenerSuite + modify a few comments 862e752 [Andrew Or] Merge branch 'master' of github.com:apache/spark into event-log-tests e0ba2f8 [Andrew Or] Fix test failures caused by race condition in processing/mutating events b990453 [Andrew Or] ReplayListenerBus suite - tests do not all pass yet ab66a84 [Andrew Or] Tests for FileLogger + delete file after tests 187bb25 [Andrew Or] Formatting and renaming variables 769336f [Andrew Or] Merge branch 'master' of github.com:apache/spark into event-log-tests 5d38ffe [Andrew Or] Clean up EventLoggingListenerSuite + add comments e12f4b1 [Andrew Or] Preliminary tests for EventLoggingListener (need major cleanup) (cherry picked from commit 394d8cb) Signed-off-by: Patrick Wendell <[email protected]>
pwendell
pushed a commit
to pwendell/spark
that referenced
this pull request
May 12, 2014
SPARK-1076: [Fix apache#578] add @transient to some vals I'll try to be more careful next time. Author: Xiangrui Meng <[email protected]> Closes apache#591 and squashes the following commits: 2b4f044 [Xiangrui Meng] add @transient to prev in ZippedWithIndexRDD add @transient to seed in PartitionwiseSampledRDD
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
Modifications to Spark core are limited to exposing functionality to test files + minor style fixes. (728 / 769 lines are from tests) Author: Andrew Or <[email protected]> Closes apache#591 from andrewor14/event-log-tests and squashes the following commits: 2883837 [Andrew Or] Merge branch 'master' of github.com:apache/spark into event-log-tests c3afcea [Andrew Or] Compromise 2d5daf8 [Andrew Or] Use temp directory provided by the OS rather than /tmp 2b52151 [Andrew Or] Remove unnecessary file delete + add a comment 62010fd [Andrew Or] More cleanup (renaming variables, updating comments etc) ad2beff [Andrew Or] Clean up EventLoggingListenerSuite + modify a few comments 862e752 [Andrew Or] Merge branch 'master' of github.com:apache/spark into event-log-tests e0ba2f8 [Andrew Or] Fix test failures caused by race condition in processing/mutating events b990453 [Andrew Or] ReplayListenerBus suite - tests do not all pass yet ab66a84 [Andrew Or] Tests for FileLogger + delete file after tests 187bb25 [Andrew Or] Formatting and renaming variables 769336f [Andrew Or] Merge branch 'master' of github.com:apache/spark into event-log-tests 5d38ffe [Andrew Or] Clean up EventLoggingListenerSuite + add comments e12f4b1 [Andrew Or] Preliminary tests for EventLoggingListener (need major cleanup)
gzm55
pushed a commit
to MediaV/spark
that referenced
this pull request
Jul 17, 2014
SPARK-1076: [Fix apache#578] add @transient to some vals I'll try to be more careful next time. Author: Xiangrui Meng <[email protected]> Closes apache#591 and squashes the following commits: 2b4f044 [Xiangrui Meng] add @transient to prev in ZippedWithIndexRDD add @transient to seed in PartitionwiseSampledRDD Conflicts: core/src/main/scala/org/apache/spark/rdd/PartitionwiseSampledRDD.scala
bzhaoopenstack
pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
update `status-url` to the specified place.
RolatZhang
pushed a commit
to RolatZhang/spark
that referenced
this pull request
Feb 24, 2023
…e#591) * Revert "Revert "[SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2" (apache#565)" This reverts commit 197dc34. * Revert "minor fix ut" This reverts commit 6ef82b4.
RolatZhang
pushed a commit
to RolatZhang/spark
that referenced
this pull request
Dec 8, 2023
…e#591) * Revert "Revert "[SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2" (apache#565)" This reverts commit 197dc34. * Revert "minor fix ut" This reverts commit 6ef82b4.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Modifications to Spark core are limited to exposing functionality to test files + minor style fixes.
(728 / 769 lines are from tests)