Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator. #20920

Closed
wants to merge 1 commit into from
Closed

[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator. #20920

wants to merge 1 commit into from

Conversation

jiangxb1987
Copy link
Contributor

What changes were proposed in this pull request?

Address #20449 (comment), If resultIter is already a InterruptibleIterator, don't double wrap it.

How was this patch tested?

Existing tests.

@jiangxb1987
Copy link
Contributor Author

cc @cloud-fan @advancedxy

advancedxy
advancedxy approved these changes Mar 28, 2018
View reviewed changes
Copy link
Contributor

@advancedxy advancedxy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sorry for didn't send this by myself. I am quite occupied.

@SparkQA
Copy link

SparkQA commented Mar 28, 2018

Test build #88664 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Mar 29, 2018

Test build #88685 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987
Copy link
Contributor Author

retest this please

@cloud-fan
Copy link
Contributor

LGTM

@jerryshao
Copy link
Contributor

LGTM.

@SparkQA
Copy link

SparkQA commented Mar 29, 2018

Test build #88691 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jerryshao
Copy link
Contributor

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Mar 29, 2018

Test build #88694 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails from timeout after a configured wait of `300m`.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Mar 29, 2018

Test build #88707 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

retest this please

@SparkQA
Copy link

SparkQA commented Mar 30, 2018

Test build #88735 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jerryshao
Copy link
Contributor

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Mar 30, 2018

Test build #88744 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987
Copy link
Contributor Author

retest this please

@jiangxb1987
Copy link
Contributor Author

Jenkins is pretty unstable recently.

@SparkQA
Copy link

SparkQA commented Mar 30, 2018

Test build #88747 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Mar 30, 2018

Test build #88754 has finished for PR 20920 at commit 35ecbf9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@asfgit asfgit closed this in 529f847 Mar 31, 2018
jiangxb1987 added a commit to jiangxb1987/spark that referenced this pull request Mar 31, 2018
@jiangxb1987
[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator.
e9a661f 
## What changes were proposed in this pull request?

Address apache#20449 (comment), If `resultIter` is already a `InterruptibleIterator`, don't double wrap it.

## How was this patch tested?
Existing tests.

Author: Xingbo Jiang <[email protected]>

Closes apache#20920 from jiangxb1987/SPARK-23040.
@jiangxb1987 jiangxb1987 mentioned this pull request Mar 31, 2018
Closed
asfgit pushed a commit that referenced this pull request Apr 1, 2018
@advancedxy @cloud-fan
[SPARK-23040][BACKPORT][CORE] Returns interruptible iterator for shuf…
f1f10da 
…fle reader

Backport #20449 and #20920 to branch-2.3

---

## What changes were proposed in this pull request?
Before this commit, a non-interruptible iterator is returned if aggregator or ordering is specified.
This commit also ensures that sorter is closed even when task is cancelled(killed) in the middle of sorting.

## How was this patch tested?
Add a unit test in JobCancellationSuite

Author: Xianjin YE <[email protected]>
Author: Xingbo Jiang <[email protected]>

Closes #20954 from jiangxb1987/SPARK-23040-2.3.
mshtelma pushed a commit to mshtelma/spark that referenced this pull request Apr 5, 2018
@jiangxb1987
[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator.
5152b3c 
## What changes were proposed in this pull request?

Address apache#20449 (comment), If `resultIter` is already a `InterruptibleIterator`, don't double wrap it.

## How was this patch tested?
Existing tests.

Author: Xingbo Jiang <[email protected]>

Closes apache#20920 from jiangxb1987/SPARK-23040.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@advancedxy advancedxy advancedxy approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jiangxb1987 @SparkQA @cloud-fan @jerryshao @advancedxy