[SPARK-2912] [Spark QA] Include commit hash in Spark QA messages #1816

nchammas · 2014-08-06T22:29:49Z

You can find the discussion that motivated this PR here.

As described in SPARK-2912, the goal of this PR (and related ones to come) is to include useful detail in Spark QA's messages that are intended to make a committer's job easier to do.

Since this work depends on Jenkins, I cannot test this locally. Hence, I will be iterating via this PR.

Notes:

This is a duplicate of a previous PR, without the extraneous commits.
This PR also resolves an issue targeted by another open PR.

merge upstream changes

SparkQA · 2014-08-06T22:34:31Z

QA tests have started for PR 1816 at commit hash 0f3ee1c.
By the way, sha1 is origin/pr/1816/merge. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18068/consoleFull

SparkQA · 2014-08-06T22:34:33Z

QA results for PR 1816:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18068/consoleFull

nchammas · 2014-08-06T22:53:18Z

By the way, sha1 is origin/pr/1816/merge.

This is a debugging statement.

@pwendell @rxin How does $sha1 change when there is a merge conflict? Or is it $ghprbActualCommit that changes? I'm not clear on how those variables are populated.

SparkQA · 2014-08-07T14:34:27Z

QA tests have started for PR 1816 at commit hash d975a23.
By the way, sha1 is origin/pr/1816/merge. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18122/consoleFull

SparkQA · 2014-08-07T14:34:28Z

QA results for PR 1816:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18122/consoleFull

SparkQA · 2014-08-07T15:04:24Z

QA tests have started for PR 1816 at commit 113f4af. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18123/consoleFull

SparkQA · 2014-08-07T15:04:26Z

QA results for PR 1816:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18123/consoleFull

SparkQA · 2014-08-07T15:34:22Z

QA tests have started for PR 1816 at commit 3ac99bf. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18124/consoleFull

SparkQA · 2014-08-07T15:34:24Z

QA results for PR 1816:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18124/consoleFull

nchammas · 2014-08-07T15:52:03Z

Jenkins, giddy on up and retest this please.

SparkQA · 2014-08-07T15:54:23Z

QA tests have started for PR 1816 at commit 3ac99bf. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18125/consoleFull

SparkQA · 2014-08-07T15:54:24Z

QA results for PR 1816:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18125/consoleFull

nchammas · 2014-08-07T17:49:07Z

Jenkinsmeister, retest this please.

SparkQA · 2014-08-07T17:54:25Z

QA tests have started for PR 1816 at commit 968e033. View progress.
* This patch merges cleanly.

SparkQA · 2014-08-07T17:54:27Z

QA results for PR 1816:
- This patch PASSES unit tests.
- This patch merges cleanly
* This patch adds no public classes

For more information see the test ouptut.

nchammas · 2014-08-07T18:16:46Z

Jenkins, touch your head, shoulders, knees, and toes, and then retest this please.

SparkQA · 2014-08-07T18:19:23Z

QA tests have started for PR 1816 at commit 481190c. View progress.

This patch merges cleanly.

SparkQA · 2014-08-07T18:19:25Z

QA results for PR 1816:

This patch PASSES unit tests.
This patch merges cleanly
This patch adds no public classes

For more information see the test ouptut.

SparkQA · 2014-08-07T20:19:28Z

QA tests have started for PR 1816 at commit 8561227. View progress.

This patch merges cleanly.

SparkQA · 2014-08-07T20:19:30Z

QA results for PR 1816 at commit 8561227:

This patch PASSES unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class StandardScaler(withMean: Boolean, withStd: Boolean) extends Logging {
For more information see the test ouptut.

SparkQA · 2014-08-07T20:54:28Z

QA tests have started for PR 1816 at commit a7d888e. View progress.

This patch merges cleanly.

SparkQA · 2014-08-07T20:54:30Z

QA results for PR 1816 at commit a7d888e:

This patch PASSES unit tests.
This patch merges cleanly.
This patch adds no public classes.

For more information see the test ouptut.

SparkQA · 2014-08-07T21:34:27Z

QA tests have started for PR 1816 at commit cbcdb61. View progress.

This patch merges cleanly.

SparkQA · 2014-08-07T21:34:28Z

QA results for PR 1816 at commit cbcdb61:

This patch PASSES unit tests.
This patch merges cleanly.
This patch adds no public classes.

For more information see the test ouptut.

SparkQA · 2014-08-07T21:54:28Z

QA tests have started for PR 1816 at commit 8eff1d3. View progress.

This patch merges cleanly.

SparkQA · 2014-08-07T21:54:30Z

QA results for PR 1816 at commit 8eff1d3:

This patch PASSES unit tests.
This patch merges cleanly.
This patch adds no public classes.

For more information see the test ouptut.

SparkQA · 2014-08-07T22:14:28Z

QA tests have started for PR 1816 at commit 391382b. View progress.

This patch merges cleanly.

SparkQA · 2014-08-07T22:24:26Z

QA tests have started for PR 1816 at commit b018df6. View progress.

This patch merges cleanly.

SparkQA · 2014-08-07T22:59:42Z

QA tests have started for PR 1816 at commit dae8407. View progress.

This patch merges cleanly.

SparkQA · 2014-08-11T20:39:42Z

QA tests have finished for PR 1816 at commit f1996f5.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DeleteMeNow()
- class DeleteMeToo(name: String = "I have quotes in me")

SparkQA · 2014-08-11T22:39:53Z

QA tests have started for PR 1816 at commit 13005b6.

This patch merges cleanly.

SparkQA · 2014-08-11T22:39:54Z

QA tests have finished for PR 1816 at commit 13005b6.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ClientActor extends Actor with ActorLogReceive with Logging
- class DeleteMeNow()
- class DeleteMeToo(name: String = "I have quotes in me")
- class DriverActor(sparkProperties: Seq[(String, String)]) extends Actor with ActorLogReceive
- * class BlockManagerMasterActor extends Actor with ActorLogReceive with Logging

nchammas · 2014-08-11T22:57:19Z

Regarding this message, sometimes I see this strangeness where git diff picks up changes that are not part of this PR. In any case, I've taken the feedback to fix the apparent formatting problem.

SparkQA · 2014-08-11T22:59:55Z

QA tests have started for PR 1816 at commit 4d87c7e.

This patch merges cleanly.

SparkQA · 2014-08-11T22:59:56Z

QA tests have finished for PR 1816 at commit 4d87c7e.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ClientActor extends Actor with ActorLogReceive with Logging
- class DeleteMePrequel()
- class DeleteMeNow()
- class DeleteMeToo(name: String = "I have quotes in me")
- class DeleteMeThree(sparkProperties: Seq[(String, String)]) extends Actor with ActorLogReceive
- class DeleteMeFour extends Actor with ActorLogReceive with Logging
- class DriverActor(sparkProperties: Seq[(String, String)]) extends Actor with ActorLogReceive
- * class BlockManagerMasterActor extends Actor with ActorLogReceive with Logging

nchammas · 2014-08-11T23:29:01Z

Jenkins, retest this please.

SparkQA · 2014-08-11T23:34:46Z

QA tests have started for PR 1816 at commit 41cb680.

This patch merges cleanly.

SparkQA · 2014-08-11T23:34:46Z

QA tests have finished for PR 1816 at commit 41cb680.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DeleteMePrequel()
- class DeleteMeNow()
- class DeleteMeToo(name: String = "I have quotes in me")

SparkQA · 2014-08-11T23:39:48Z

QA tests have started for PR 1816 at commit e029e02.

This patch merges cleanly.

SparkQA · 2014-08-12T00:30:43Z

QA tests have finished for PR 1816 at commit e029e02.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ClientActor extends Actor with ActorLogReceive with Logging
- class DriverActor(sparkProperties: Seq[(String, String)]) extends Actor with ActorLogReceive

SparkQA · 2014-08-12T01:14:47Z

QA tests have started for PR 1816 at commit a20fcad.

This patch merges cleanly.

SparkQA · 2014-08-12T01:14:47Z

QA tests have finished for PR 1816 at commit a20fcad.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

This commit makes 2 major changes to the Jenkins GitHub hook: 1. Include commit hashes in all messages posted to GitHub. This makes it clear what set of code changes a given test cycle is for. 2. Reorganize the code so it’s easier to maintain. Changes made as part of the code reorg include: * Cleanly capture and report on errors when calling the GitHub API. * Divide the code into clear blocks: pre-test; test; and post-test

SparkQA · 2014-08-12T01:34:46Z

QA tests have started for PR 1816 at commit c1be644.

This patch merges cleanly.

SparkQA · 2014-08-12T02:25:56Z

QA tests have finished for PR 1816 at commit c1be644.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

nchammas · 2014-08-12T02:30:35Z

@pwendell @rxin - After lots of futzing around and some gold plating, I can say this PR is ready for another review. Apologies for how long this took.

Highlights:

This PR also resolves the issue targeted by this open PR.
Here is an example of the output when new classes are added.
- The filters for that now handle many more cases than before, like new classes that are commented out.
- The filters are also documented using an epic bash hack.

rxin · 2014-08-14T00:50:27Z

This is really cool. I will defer to @pwendell for merging this.

rxin · 2014-08-14T00:50:54Z

In the future maybe we should rewrite this in Python so the community can maintain it (bash is great until it gets too complicated).

nchammas · 2014-08-14T01:02:11Z

Agreed. As part of this PR, I originally wanted to include functionality to update older messages with notes like "New commits have been added since this test cycle", but it seemed like quite too much to do with bash. function get_jq () is a leftover from that attempt, for example.

You can find the [discussion that motivated this PR here](http://mail-archives.apache.org/mod_mbox/spark-dev/201408.mbox/%3CCABPQxssy0ri2QAz=cc9Tx+EXYWARm7pNcVm8apqCwc-esLbO4Qmail.gmail.com%3E). As described in [SPARK-2912](https://issues.apache.org/jira/browse/SPARK-2912), the goal of this PR (and related ones to come) is to include useful detail in Spark QA's messages that are intended to make a committer's job easier to do. Since this work depends on Jenkins, I cannot test this locally. Hence, I will be iterating via this PR. Notes: * This is a duplicate of a [previous PR](apache#1811), without the extraneous commits. * This PR also resolves an issue targeted by [another open PR](apache#1809). Closes apache#1809. Author: Nicholas Chammas <[email protected]> Author: nchammas <[email protected]> Closes apache#1816 from nchammas/master and squashes the following commits: c1be644 [Nicholas Chammas] [SPARK-2912] include commit hash in messages 8f641ac [nchammas] Merge pull request apache#7 from apache/master

Merge pull request #7 from apache/master

8f641ac

merge upstream changes

nchammas changed the title ~~[SPARK-2912] [Spark QA] [WIP] Include commit hash in Spark QA messages~~ [SPARK-2912] [Spark QA] Include commit hash in Spark QA messages Aug 12, 2014

asfgit closed this in 500f84e Aug 15, 2014

nchammas mentioned this pull request Aug 30, 2014

Add new PR categories for Docs, Spark QA, and EC2 databricks/spark-pr-dashboard#10

Closed

[SPARK-2912] [Spark QA] Include commit hash in Spark QA messages #1816

[SPARK-2912] [Spark QA] Include commit hash in Spark QA messages #1816

Conversation

nchammas commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

nchammas commented Aug 6, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

nchammas commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

nchammas commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

nchammas commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

nchammas commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

nchammas commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 12, 2014

SparkQA commented Aug 12, 2014

SparkQA commented Aug 12, 2014

SparkQA commented Aug 12, 2014

SparkQA commented Aug 12, 2014

nchammas commented Aug 12, 2014

rxin commented Aug 14, 2014

rxin commented Aug 14, 2014

nchammas commented Aug 14, 2014