Skip to content
This repository has been archived by the owner on May 9, 2024. It is now read-only.

[SPARK-8977][Streaming] Defines the RateEstimator interface, and implements the ReceiverRateController #16

Closed
wants to merge 1 commit into from
Closed

[SPARK-8977][Streaming] Defines the RateEstimator interface, and implements the ReceiverRateController #16

wants to merge 1 commit into from

Conversation

huitseeker
Copy link

Depends on #15.

@huitseeker huitseeker force-pushed the SPARK-8977 branch 3 times, most recently from 758b9bc to 397b8b7 Compare July 14, 2015 14:42
This was referenced Jul 14, 2015
Closed
Closed
@huitseeker huitseeker force-pushed the SPARK-8977 branch 4 times, most recently from 6cd2b46 to 78219dd Compare July 15, 2015 00:34
@typesafe-tools
Copy link
Collaborator

Refer to this link for build results (access rights to CI server needed):
https://ci.typesafe.com/job/ghprb-spark-multi-conf/42/

Build Log
last 10 lines

[...truncated 16 lines...]
 > git rev-parse refs/remotes/origin/origin/pr/16/merge^{commit} # timeout=10
Checking out Revision 2e2fbad59abfff947c010f931d700d8753634a15 (refs/remotes/origin/pr/16/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2e2fbad59abfff947c010f931d700d8753634a15
First time build. Skipping changelog.
Triggering ghprb-spark-multi-conf » Spark-Ora-JDK7-PV,2.10
ghprb-spark-multi-conf » Spark-Ora-JDK7-PV,2.10 completed with result FAILURE
Notifying upstream projects of job completion
Setting status of 8a274e7bdf692c9bb4e41d924521e8b455de8169 to FAILURE with url http://ci.typesafe.com/job/ghprb-spark-multi-conf/42/ and message: Merged build finished.

Test FAILed.

dragos
dragos reviewed Jul 15, 2015
View reviewed changes
streaming/src/main/scala/org/apache/spark/streaming/scheduler/RateController.scala

protected def publish(rate: Long): Unit = ()

private def computeAndPublishExecutionContext = ExecutionContext.fromExecutorService(
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be a val, or maybe even an implicit val?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would pose a serializability problem.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@transient would fix it. Wouldn't the current code create a new thread on each rate computation?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, yes, you're right !

@huitseeker
Copy link
Author

unrelated SQL tests failing: https://ci.typesafe.com/job/ghprb-spark-multi-conf/label=Spark-Ora-JDK7-PV,scala_version=2.10/42/console

@huitseeker huitseeker force-pushed the SPARK-8977 branch 3 times, most recently from 99a40fe to 8479165 Compare July 15, 2015 11:15
@typesafe-tools
Copy link
Collaborator

Refer to this link for build results (access rights to CI server needed):
https://ci.typesafe.com/job/ghprb-spark-multi-conf/44/

Build Log
last 10 lines

[...truncated 11 lines...]
 > git rev-parse refs/remotes/origin/origin/pr/16/merge^{commit} # timeout=10
Checking out Revision a32021467c61f16cc5b8181b85ab62503bf3852e (refs/remotes/origin/pr/16/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a32021467c61f16cc5b8181b85ab62503bf3852e
First time build. Skipping changelog.
Triggering ghprb-spark-multi-conf » Spark-Ora-JDK7-PV,2.10
ghprb-spark-multi-conf » Spark-Ora-JDK7-PV,2.10 completed with result FAILURE
Notifying upstream projects of job completion
Setting status of 758b9bc086159548838c9ff31e6d7ef146e77c21 to FAILURE with url http://ci.typesafe.com/job/ghprb-spark-multi-conf/44/ and message: Merged build finished.

Test FAILed.

@huitseeker
Copy link
Author

https://ci.typesafe.com/job/ghprb-spark-multi-conf/label=Spark-Ora-JDK7-PV,scala_version=2.10/44/console
<< unrelated SQL

@typesafe-tools
Copy link
Collaborator

Refer to this link for build results (access rights to CI server needed):
https://ci.typesafe.com/job/ghprb-spark-multi-conf/45/

Build Log
last 10 lines

[...truncated 11 lines...]
 > git rev-parse refs/remotes/origin/origin/pr/16/merge^{commit} # timeout=10
Checking out Revision a32021467c61f16cc5b8181b85ab62503bf3852e (refs/remotes/origin/pr/16/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a32021467c61f16cc5b8181b85ab62503bf3852e
First time build. Skipping changelog.
Triggering ghprb-spark-multi-conf » Spark-Ora-JDK7-PV,2.10
ghprb-spark-multi-conf » Spark-Ora-JDK7-PV,2.10 completed with result FAILURE
Notifying upstream projects of job completion
Setting status of 397b8b7b3e98904740df3f38d1b81b2508dc74b8 to FAILURE with url http://ci.typesafe.com/job/ghprb-spark-multi-conf/45/ and message: Merged build finished.

Test FAILed.

@huitseeker
Copy link
Author

https://ci.typesafe.com/job/ghprb-spark-multi-conf/label=Spark-Ora-JDK7-PV,scala_version=2.10/45/console << Unrelated SQL again

@typesafe-tools
Copy link
Collaborator

Refer to this link for build results (access rights to CI server needed):
https://ci.typesafe.com/job/ghprb-spark-multi-conf/51/

Build Log
last 10 lines

[...truncated 11 lines...]
 > git rev-parse refs/remotes/origin/origin/pr/16/merge^{commit} # timeout=10
Checking out Revision 5abf73eb27c412e0035006aa33910e91e6e593b3 (refs/remotes/origin/pr/16/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5abf73eb27c412e0035006aa33910e91e6e593b3
First time build. Skipping changelog.
Triggering ghprb-spark-multi-conf » Spark-Ora-JDK7-PV,2.10
ghprb-spark-multi-conf » Spark-Ora-JDK7-PV,2.10 completed with result FAILURE
Notifying upstream projects of job completion
Setting status of 02b6c4b68024365a5bfbd27f61bde6a33a4b6c4f to FAILURE with url http://ci.typesafe.com/job/ghprb-spark-multi-conf/51/ and message: Merged build finished.

Test FAILed.

@typesafe-tools
Copy link
Collaborator

Refer to this link for build results (access rights to CI server needed):
https://ci.typesafe.com/job/ghprb-spark-multi-conf/55/

Build Log
last 10 lines

[...truncated 11 lines...]
 > git rev-parse refs/remotes/origin/origin/pr/16/merge^{commit} # timeout=10
Checking out Revision 5abf73eb27c412e0035006aa33910e91e6e593b3 (refs/remotes/origin/pr/16/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5abf73eb27c412e0035006aa33910e91e6e593b3
First time build. Skipping changelog.
Triggering ghprb-spark-multi-conf » Spark-Ora-JDK7-PV,2.10
ghprb-spark-multi-conf » Spark-Ora-JDK7-PV,2.10 completed with result ABORTED
Notifying upstream projects of job completion
Setting status of f3d7ac96e036777bed252a064113721f3074409e to FAILURE with url http://ci.typesafe.com/job/ghprb-spark-multi-conf/55/ and message: Merged build finished.

Test FAILed.

@dragos dragos force-pushed the topic/streaming-bp/dynamic-rate branch from 13ada97 to 0c51959 Compare July 20, 2015 14:45
@dragos dragos closed this Sep 9, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@huitseeker @typesafe-tools @dragos