Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-35362][SQL] Update null count in the column stats for UNION operator stats estimation #32494

Closed
wants to merge 16 commits into from
Closed

[SPARK-35362][SQL] Update null count in the column stats for UNION operator stats estimation #32494

wants to merge 16 commits into from

Conversation

shahidki31
Copy link
Contributor

What changes were proposed in this pull request?

Updating column stats for Union operator stats estimation

Why are the changes needed?

This is a followup PR to update the null count also in the Union stats operator estimation. #30334

Does this PR introduce any user-facing change?

No

How was this patch tested?

Updated UTs, manual testing

@github-actions github-actions bot added the SQL label May 10, 2021
@shahidki31 shahidki31 changed the title [SPARK-35362][SQL]Update null count in the column stats for UNION stats estimation [SPARK-35362][SQL]Update null count in the column stats for UNION operator stats estimation May 10, 2021
@SparkQA
Copy link

SparkQA commented May 10, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42864/

@SparkQA
Copy link

SparkQA commented May 10, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42864/

@shahidki31 shahidki31 changed the title [SPARK-35362][SQL]Update null count in the column stats for UNION operator stats estimation [Minor][SPARK-35362][SQL]Update null count in the column stats for UNION operator stats estimation May 10, 2021
@SparkQA
Copy link

SparkQA commented May 11, 2021

Test build #138342 has finished for PR 32494 at commit 56ea3d8.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@shahidki31
Copy link
Contributor Author

Retest this please

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42873/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42873/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Test build #138351 has finished for PR 32494 at commit 8e02f19.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon HyukjinKwon changed the title [Minor][SPARK-35362][SQL]Update null count in the column stats for UNION operator stats estimation [SPARK-35362][SQL] Update null count in the column stats for UNION operator stats estimation May 11, 2021
@HyukjinKwon
Copy link
Member

cc @maropu and @wangyum

@HyukjinKwon
Copy link
Member

cc @wzhfy too

@shahidki31 shahidki31 force-pushed the shahid/updateNullCountForUnion branch from 8e02f19 to a1b1d5f Compare May 11, 2021 19:30
@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42922/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42922/

@SparkQA
Copy link

SparkQA commented May 12, 2021

Test build #138400 has finished for PR 32494 at commit a1b1d5f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 12, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42950/

@SparkQA
Copy link

SparkQA commented May 12, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42950/

@SparkQA
Copy link

SparkQA commented May 12, 2021

Test build #138429 has finished for PR 32494 at commit d876e7b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 12, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42980/

@SparkQA
Copy link

SparkQA commented May 12, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42980/

@SparkQA
Copy link

SparkQA commented May 18, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43172/

@SparkQA
Copy link

SparkQA commented May 18, 2021

Test build #138647 has finished for PR 32494 at commit bcd628b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 18, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43172/

@SparkQA
Copy link

SparkQA commented May 18, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43183/

@SparkQA
Copy link

SparkQA commented May 18, 2021

Test build #138653 has finished for PR 32494 at commit 74d73a3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 18, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43183/

maropu
maropu reviewed May 18, 2021
View reviewed changes
Copy link
Member

@maropu maropu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks almost good and thank you for this work, @shahidki31

@SparkQA
Copy link

SparkQA commented May 18, 2021

Test build #138662 has finished for PR 32494 at commit 4c73828.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 18, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43212/

@SparkQA
Copy link

SparkQA commented May 18, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43212/

@SparkQA
Copy link

SparkQA commented May 19, 2021

Test build #138691 has finished for PR 32494 at commit 3d38208.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 19, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43218/

@SparkQA
Copy link

SparkQA commented May 19, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43218/

@SparkQA
Copy link

SparkQA commented May 19, 2021

Test build #138697 has finished for PR 32494 at commit 3afaf32.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member

maropu commented May 19, 2021

Looks fine otherwise.

@SparkQA
Copy link

SparkQA commented May 19, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43226/

@SparkQA
Copy link

SparkQA commented May 19, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43226/

@maropu maropu closed this in 1214213 May 19, 2021
@maropu
Copy link
Member

maropu commented May 19, 2021

Thank you, @shahidki31 . Merged to master.

@shahidki31
Copy link
Contributor Author

Thanks a lot @maropu for the reviews.

@shahidki31 shahidki31 deleted the shahid/updateNullCountForUnion branch May 19, 2021 12:38
@SparkQA
Copy link

SparkQA commented May 19, 2021

Test build #138705 has finished for PR 32494 at commit 06fbbec.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maropu maropu maropu approved these changes

Assignees
No one assigned
Labels
SQL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@shahidki31 @SparkQA @HyukjinKwon @maropu