hms-mirror

Added command line flag -cine|--create-if-not-exist to alter create statements. https://github.com/cloudera-labs/hms-mirror#create-external-table-if-not-exists-option

Support for non-apache jdbc drivers (Cloudera JDBC Driver). #70

Changed Connection Pooling Libraries for Datasources to Hikari-CP. #71

Adjusted screen reporting to show initialization of tables and metadata before processing starts.

Improved Testing Framework.

hms-mirror

Stats collection criteria were added to avoid getting stats when not needed. Add skipStatsCollection option to avoid when using a file system that isn't great at reporting this. #68

Allow Multiple Jars in jarFile: #67

Adjust CREATE to CREATE IF NOT EXISTS with option: #66

Add 'evaluate partition location' -epl option to handle NON-STANDARD partition locations/names: #64

Extend STORAGE_MIGRATION support to use -dc: #65

Fixed Progress Indicator. Add error count to return code of application #69

hms-mirror

Non-Standard Partition: #61

Ability to turn off/on stats collection: #59

Some additional connection pooling check/catches.

hms-mirror

Max Reducers setting, when needed, was a double under certain conditions. Cast to INT to allow Hive to set value in session.

hms-mirror

handle data-size issues with stats when not available.

added more traps for hdpHive3 (lack of features check) hive.optimize.sort.dynamic.partition.threshold.

hms-mirror

For extremely large tables with a lot of partitions, we fixed the max reducer calculations to match the need based on the distribution.

hms-mirror

Fixed some casting issues while setting dynamic partitions and max reducers.

hms-mirror

Features:

• Auto-Tuning (-at)- Introduction of basic stats regarding file counts/sizes for large tables. We'll make adjustments to DISTRIBUTE BY and Tez Groupings to provide more efficient/balanced migrations with better/more optimized file sizes after migration for migrations using SQL. #53
• AAdditional table filters (-tfs|--table-filter-size-limit and -tfp|--table-filter-partition-count-limit) that check a tables data size and partition count limits can also be applied to narrow the range of tables you'll process. #55
• Add property to tables migrated with "STORAGE_MIGRATION" to identify and filter them out from future runs. #56
• -cto|--compress-text-output option and additional session level settings using basic stats.
• HDP3 scenario that doesn't support MANAGEDLOCATION element in database properties. #52

Fixes:

• AVRO Schema Only Fix.. #58
• Cleanup messaging around legacy config settings.
• Fix/Added dbRegEx command line parameter: #57

NOTE: Configuration Breaking Change. If you see note about A configuration element is no longer valid, progress. Please remove the element from the configuration yaml and try again. with Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "tblRegEx", please remove the properties dbRegEx, tblRegEx and tblExcludeRegEx from the config yaml.

hms-mirror

Features:

• Auto-Tuning - Introduction of basic stats regarding file counts/sizes for large tables. We'll make adjustments to DISTRIBUTE BY and Tez Groupings to provide more efficient/balanced migrations with better/more optimized file sizes after migration for migrations using SQL. #53
• AAdditional table filters (-tfs|--table-filter-size-limit and -tfp|--table-filter-partition-count-limit) that check a tables data size and partition count limits can also be applied to narrow the range of tables you'll process. #55
• Add property to tables migrated with "STORAGE_MIGRATION" to identify and filter them out from future runs. #56
• -cto|--compress-text-output option and additional session level settings using basic stats.
• HDP3 scenario that doesn't support MANAGEDLOCATION element in database properties. #52

Fixes:

• AVRO Schema Only Fix.. #58
• Cleanup messaging around legacy config settings.
• Fix/Added dbRegEx command line parameter: #57

NOTE: Configuration Breaking Change. If you see note about A configuration element is no longer valid, progress. Please remove the element from the configuration yaml and try again. with Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "tblRegEx", please remove the properties dbRegEx, tblRegEx and tblExcludeRegEx from the config yaml.

hms-mirror

Support for HDP Hive 3 anomalies regarding locations.

#52