Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-34518][SQL] Rename AlterTableRecoverPartitionsCommand to RepairTableCommand #31635

Closed
wants to merge 2 commits into from

Conversation

MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Feb 24, 2021

What changes were proposed in this pull request?

Rename the execution node AlterTableRecoverPartitionsCommand for the commands:

  • MSCK REPAIR TABLE table [{ADD|DROP|SYNC} PARTITIONS]
  • ALTER TABLE table RECOVER PARTITIONS

to RepairTableCommand.

Why are the changes needed?

  1. After the PR [SPARK-31891][SQL] Support MSCK REPAIR TABLE .. [{ADD|DROP|SYNC} PARTITIONS] #31499, ALTER TABLE table RECOVER PARTITIONS is equal to MSCK REPAIR TABLE table ADD PARTITIONS. And mapping of the generic command MSCK REPAIR TABLE to the more specific execution node AlterTableRecoverPartitionsCommand can confuse devs in the future.
  2. ALTER TABLE table RECOVER PARTITIONS does not support any options/extensions. So, additional parameters enableAddPartitions and enableDropPartitions in AlterTableRecoverPartitionsCommand confuse as well.

Does this PR introduce any user-facing change?

No because this is internal API.

How was this patch tested?

By running the existing test suites:

$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *AlterTableRecoverPartitionsSuite"
$ build/sbt "test:testOnly *AlterTableRecoverPartitionsParserSuite"
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *MsckRepairTableSuite"
$ build/sbt "test:testOnly *MsckRepairTableParserSuite"

@github-actions github-actions bot added the SQL label Feb 24, 2021
@SparkQA
Copy link

SparkQA commented Feb 24, 2021

Test build #135413 has finished for PR 31635 at commit 1336293.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MaxGekk
Copy link
Member Author

MaxGekk commented Feb 24, 2021

@dongjoon-hyun @cloud-fan @HyukjinKwon Are you ok with the renaming?

@@ -600,11 +600,11 @@ case class PartitionStatistics(numFiles: Int, totalSize: Long)
* MSCK REPAIR TABLE table [{ADD|DROP|SYNC} PARTITIONS];
* }}}
*/
case class AlterTableRecoverPartitionsCommand(
case class RepairTableCommand(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit reluctant to rename v1 commands, as they are legacy and have been there for many years.

Copy link
Member Author

@MaxGekk MaxGekk Feb 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but this is non-public API. We have already extended AlterTableRecoverPartitionsCommand in #31499. After that, it became incompatible by sources and binary with the previous version.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And how about the logical nodes that were massively replaced in https://issues.apache.org/jira/browse/SPARK-29900. They are legacy and have been there for many years too, aren't they?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the point is that we might better focus on checking V2 instead of V1 commands because eventually V2 commands should replace V1 commands in the future. If this is likely only the one, I think it's fine. If there are a lot of such instances, I wouldn't change for now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is likely only the one, I think it's fine.

Yep, only this for now.

Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm okay with this but let me leave it to @cloud-fan.

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in c56af69 Feb 25, 2021
@dongjoon-hyun
Copy link
Member

+1, LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants