Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When transferring ACID tables, allow for non-optimization options #23

Closed
dstreev opened this issue Jan 18, 2023 · 2 comments
Closed

When transferring ACID tables, allow for non-optimization options #23

dstreev opened this issue Jan 18, 2023 · 2 comments
Assignees
@dstreev
Labels
enhancement New feature or request
Milestone
1.5.4.3

Comments

@dstreev
Copy link
Collaborator

dstreev commented Jan 18, 2023

When the source table is well compressed and the files are large, a map only job is ok to help with the transfer. In this case, no files are combined, so the mappers will only write to the partition from which they got the source file.

Also consider setting: hive.exec.orc.split.strategy=BI to help with file organization.
@dstreev dstreev added the enhancement New feature or request label Jan 18, 2023
@dstreev dstreev self-assigned this Jan 18, 2023
@dstreev
Copy link
Collaborator Author

dstreev commented Jan 18, 2023

When the source table is organized AND the table has large files, the DISTRIBUTE BY and DYNAMIC SORTING options, which introduce a REDUCE phase will get choked up on the writes. A partition will only be written to by a single file. When we really want multiple writers.

@dstreev dstreev added this to the 1.5.4.3 milestone Mar 14, 2023
@dstreev
Copy link
Collaborator Author

dstreev commented Mar 14, 2023

Added property to CLI: -so|--skip-optimizations. This will set the property hive.optimize.sort.dynamic.partition=false and NOT add DISTRIBUTE BY to the SQL statements.

@dstreev dstreev closed this as completed Mar 14, 2023
dstreev added a commit to dstreev/hms-mirror that referenced this issue Mar 14, 2023
@dstreev
No-Purge, Skip-Optimizations, Property Overrides.
8e62734 
[Property Overrides](cloudera-labs/hms-mirror#27)
[No-Purge](cloudera-labs/hms-mirror#25)
[Skip Optimizations](cloudera-labs/hms-mirror#23)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dstreev dstreev

Labels
enhancement New feature or request
Projects
None yet
Milestone
1.5.4.3
Development

No branches or pull requests
1 participant
@dstreev