Add release notes for 0.232

[caithagoras]

Missing Release Notes

Andrii Rosa

• 2b56476 Run presto spark tests in forked VM
• e8b6042 Fix hostname related assertion failure in PrestoSparkQueryRunner

Da Cheng

• Update Kafka connector to 2.3.1 #13709 Update Kafka connector to 2.3.1 (Merged by: Zhenxiao Luo)

Haoyuan Li

• Add Alluxio Hive metastore #13986 Add Alluxio Hive metastore (Merged by: James Sun)

YuBin Li

• Support kerberos authentication for Kudu connector #14007 Support kerberos authentication for Kudu connector (Merged by: Wenlei Xie)

qqibrow

• Fix Elasticsearch connector documentation #14017 Fix Elasticsearch connector documentation (Merged by: Wenlei Xie)

sujay-jain

• 7700a59 Revert "Improve error message for the array_agg function"

wuyunfeng01

• Promote Elasticsearch scan performance with index sorting #13987 Promote Elasticsearch scan performance with index sorting (Merged by: Zhenxiao Luo)

Extracted Release Notes

• Add use-exact-partitioning-enabled session property #13354 (Author: Ariel Weisberg): Add use-exact-partitioning-enabled session property
  • Add use_exact_partitioning session property that forces repartitioning if repartitioning is possible.
• Support retries of streaming sections #13717 (Author: Rebecca Schlussel): Support retries of streaming sections
  • Add support for retrying failed stages from a materialized point instead of failing the entire query. The number of retries allowed can be configured using the configuration property max-stage-retries and session property max_stage_retries. The default value is zero. To take advantage of this feature, exchange_materialization_strategy must be set to 'ALL'.
  • Add configuration property use-legacy-scheduler and session property use_legacy_scheduler to use a version of the query scheduler from before refactorings to enable full stage retries. The default value is false. This is a temporary property to provide an easy way to roll back in case of bugs in the new scheduler. This property will be removed in a couple releases once we have confidence in the stability of the new scheduler.
• Support ALTER FUNCTION #13799 (Author: Leiqing Cai): Support ALTER FUNCTION
  • Add support for ALTER FUNCTION.
• Support internal communication with thrift #13894 (Author: James Sun): Support internal communication with thrift
  • Allow Presto nodes to shuffle data with Thrift protocol. Use config internal-communication.task-communication-protocol to control between HTTP and Thrift.
  • Allow Presto nodes to announce state with Thrift protocol. Use config internal-communication.server-info-communication-protocol to control between HTTP and Thrift.
• Configure DB Resource Group reload frequency #13931 (Author: Swapnil Tailor): Configure DB Resource Group reload frequency
  • Making database resource group reload frequency configurable.
  • ...
• Update error code when table is dropped during query execution #13934 (Author: Vic Zhang): Update error code when table is dropped during query execution
  • When DROP TABLE query fails because the table has being dropped by other query before this query finishes, change error code name from HIVE_METASTORE_ERROR to HIVE_TABLE_DROPPED_DURING_QUERY.
• Add t-digest functions to Presto #13940 (Author: Timothy Meehan): Add t-digest functions to Presto
  • Add tdigest_agg, merge(tdigest), value_at_quantile(tdigest, quantile), values_at_quantiles(tdigest, quantiles), quantile_at_value(tdigest, quantile), quantiles_at_values(tdigest, quantile) for creating, merging, and querying t-digests.
  • IsHidden attribute on AggregationFunction and ScalarFunction removed and replaced with visibility.
• Enable listing SQL function with session property #13949 (Author: Leiqing Cai): Enable listing SQL function with session property
  • Support hiding user-defined SQL functions in SHOW FUNCTIONS with session property list_built_in_functions_only. This can also be achieved by configuration property list-built-in-functions-only, which is repurposed from list-non-built-in-functions.
• Support MySQL-based FunctionNamespaceManager #13972 (Author: Leiqing Cai): Support MySQL-based FunctionNamespaceManager
  • Add a MySQL-based function namespace manager implementation that supports creating, altering, and dropping SQL functions. (:doc:/admin/function-namespace-managers).
• Extend SHOW FUNCTIONS #13974 (Author: Leiqing Cai): Extend SHOW FUNCTIONS
  • Add support to show whether functions have variable arity in SHOW FUNCTIONS.
  • Add support to show whether functions are built-in in SHOW FUNCTIONS.
• Cache builtin function resolution result #13982 (Author: Rongrong Zhong): Cache builtin function resolution result
  • Improve built-in function resolution performance by caching function resolution results.
• Improve correctness check for queries that produce columns of RowType #13998 (Author: Igor Kruk): Improve correctness check for queries that produce columns of RowType
  • Add checks for the individual fields when validating a row column.
• Support shuffle on Hive partition columns before write #14010 (Author: Wenlei Xie): Support shuffle on Hive partition columns before write
  • Allow shuffle on partitioned columns when writing to partitioned unbucketed Hive tables. This increase the number of maximum partitions written in single query by a factor of number of total writing workers.This behavior has to be explicitly enabled by Connector session property shuffle_partitioned_columns_for_table_write. Support shuffle on Hive partition columns before write #14010.
• Upgrade slice to 0.38 #14014 (Author: James Gill): Upgrade slice to 0.38
  • Upgrade airlift/slice to 0.38.
• Add task memory related session properties  #14016 (Author: Ariel Weisberg): Add task memory related session properties
  • Added query_max_total_memory_per_node and query_max_memory_per_node session properties.
• Add statistics handling to the Alluxio metastore support #14018 (Author: Haoyuan Li): Add statistics handling to the Alluxio metastore support
  • Add statistics handling to the Alluxio metastore support.
• Improve correctness check for RowType columns #14036 (Author: Igor Kruk): Improve correctness check for RowType columns
  • Add specific validation checks for the individual fields when validating a row column.
• Account for page header in ORC output buffer #14039 (Author: Islam Ismailov): Account for page header in ORC output buffer
  • Add page header size accounting to ORC output buffer to avoid possible overflow.
• Expose Hive table properties via system table$properties table #14043 (Author: qqibrow): Expose Hive table properties via system table$properties table
  • Expose Hive table properties via system table$properties table.
• Update Alluxio version from 2.1.1 to 2.1.2 #14047 (Author: Haoyuan Li): Update Alluxio version from 2.1.1 to 2.1.2
  • Update Alluxio version from 2.1.1 to 2.1.2.
• Pinot bugs and making it work with the latest trunk pinot #14057 (Author: Devesh Agrawal): Pinot bugs and making it work with the latest trunk pinot
  • Replace config pinot.prefer-broker-queries with the inverse config pinot.forbid-broker-queries.
• Update to Joda 2.10.5 #14059 (Author: Leiqing Cai): Update to Joda 2.10.5
  • Fix an issue where DATE_TRUNC may produce incorrect results at certain timestamp in America/Sao_Paulo.

All Commits

• 2b56476 Run presto spark tests in forked VM (Andrii Rosa)
• 7700a59 Revert "Improve error message for the array_agg function" (sujay-jain)
• e8b6042 Fix hostname related assertion failure in PrestoSparkQueryRunner (Andrii Rosa)
• 44c044a Add documentation for SQL functions (Leiqing Cai)
• fbe0b3d Introduce MySql-based function namespace manager (Leiqing Cai)
• a59bef8 Introduce db based function namespace manager (Leiqing Cai)
• c2c90c0 Use Return statement as the body for SQL-invoked functions (Leiqing Cai)
• dc129d8 Require function namespace manager to bind with catalog names (Leiqing Cai)
• 56d063b Remove dependency of presto-sql-function from presto-main (Leiqing Cai)
• 2de5d7f Improve correctness check for RowType columns (Igor Kruk)
• 4c2010e Add retriedCpuTime to QueryCompletedEvent (Rebecca Schlussel)
• 6525ea9 Naming cleanup in SqlQueryScheduler (Rebecca Schlussel)
• 615feb5 Add support for retrying streaming sections (Rebecca Schlussel)
• e5eda57 Introduce LegacySqlQueryScheduler (Rebecca Schlussel)
• fe6ebcf Reorder fields and methods in SqlQueryScheduler (Rebecca Schlussel)
• 4fd684b Make FixedSourcePartitionedScheduler more thread safe (Rebecca Schlussel)
• f789c54 Introduce method for empty StageExecutionInfo (Rebecca Schlussel)
• 2e4d5a3 Remove id from StageExecutionInfo (Rebecca Schlussel)
• b84f177 Make ExecutionSchedule return StageExecutionAndScheduler (Rebecca Schlussel)
• 3185bf4 Extract SectionExecutionFactory from SqlQueryScheduler (Rebecca Schlussel)
• 08793ed Add exponential decay config to failure detector (Andrii Rosa)
• 3eef1b0 Enable failure detector in tests (Rebecca Schlussel)
• b937625 Remove unused code from TestStageExecutionStateMachine (Rebecca Schlussel)
• 85ed09a Add configuration property max-stage-retries (Rebecca Schlussel)
• e650db3 Minor refactor in SqlQueryScheduler (Rebecca Schlussel)
• 7a4a741 Move opening split source to scheduler creation time (Rebecca Schlussel)
• 4fd9aaf Log number of splits received for Presto-on-Spark task (Wenlei Xie)
• b490921 Update Airlift to 0.189 (Andrii Rosa)
• a3f9aa3 Fail with informative error message if no Pinot expression is selected (Devesh Agrawal)
• 1701b22 Avoid passing Content-Type header when Pinot request body is empty (Devesh Agrawal)
• 74ad34b Rename preferBrokerQueries to its negate format forbidBrokerQueries (Devesh Agrawal)
• b05e09e Fix bug with pinot column scan index generation (Devesh Agrawal)
• 4c7fc03 Add join pushdown predicate at the end of the exisiting predicate (Bhavani Hari)
• f638755 Allocate CPU quanta per query instead of per task (Andrii Rosa)
• 46e6c92 Update proto utilities for the Database objects (Haoyuan Li)
• 5833338 Support shuffle on Hive partition columns before write (Wenlei Xie)
• b4a186c Rename TableWriterNode#partitioningScheme to tablePartitioningScheme (Wenlei Xie)
• 5a7074b Adding LIKE operator description (Oskar Austegard)
• 47afe84 Update to Joda 2.10.5 (Leiqing Cai)
• 10893a6 Account for page header in ORC output buffer (Islam Ismailov)
• 8ac000d Add peak task memory in resource estimates (Andrés Flores)
• 1b45f49 Update alluxio version (Haoyuan Li)
• 2dd6798 Fix recoverable grouped execution eligibility (Shixuan Fan)
• 4f13800 Replace Joda-Time libraries with java.time (Ajay George)
• 05310b7 Add statistics handling to the Alluxio metastore support (Haoyuan Li)
• fc8cb11 Expose Hive table properties via system table$properties table (qqibrow)
• 0c5aa2d Improve correctness check for queries that produce columns of RowType (Igor Kruk)
• fc3d1ee Add task memory related session properties (Ariel Weisberg)
• 8eb0192 Upgrade Kafka Connector to 2.3.1 (Da Cheng)
• 1ec8138 Support kerberos authentication for Kudu connector (YuBin Li)
• 66fb17c Fix target path typo in HiveWriter (James Sun)
• a26f440 Optimize T-Digest structure for small distributions (Timothy Meehan)
• 8296be0 Move PushdownSubfields below logical optimization (Saksham Sachdev)
• 30f1db0 Cache mvnw (Tim Meehan)
• dcbbe44 Fix Elasticsearch connector documentation (Zhenxiao Luo)
• 8a1bb1b Extend SHOW FUNCTIONS to display whether functions are built-in (Leiqing Cai)
• c3b0432 Extend SHOW FUNCTIONS to display whether functions have variable arity (Leiqing Cai)
• 283e300 Fix testShowFunctions (Leiqing Cai)
• b1cda97 Upgrade slice to 0.38 (James Gill)
• c487827 Improve error message for the array_agg function (Sujay Jain)
• 6eed9b9 Support mixed case fields in Elasticsearch (Zhenxiao Luo)
• a9bbc6e Set predicate in EMPTY_TABLE_LAYOUT to none (Bhavani Hari)
• 5106dc1 Add Alluxio Hive metastore (haoyuan)
• ce5eeb7 Update error code when table is dropped during query execution (Vic Zhang)
• 823da6f Evaluate filter only once per dictionary value (Masha Basmanova)
• d5ee715 Separate filter and no-filter code paths (Masha Basmanova)
• ae5e74c Fix flaky TestRowExpressionPredicateExtractor (Ajay George)
• cd41fa5 Format comment and modify import style (wuyunfeng01)
• ab55def Promote Elasticsearch scan performance with index sorting (wuyunfeng01)
• b34278e Properly handle optimizable filter expressions (Bhavani Hari)
• fb37c6e Add t-digest functions to Presto (Tim Meehan)
• d09e4df Migrate function isHidden to new visibility enum (Tim Meehan)
• 63cf099 Allow FunctionNamespaceManager in DistributedQueryRunner (Rongrong Zhong)
• 683474a Cache builtin function resolution result (Rongrong Zhong)
• 2e96b92 Update access modifiers in BenchmarkSelectiveStreamReaders (Ying Su)
• f555aa6 Avoid Java stream in NodePartitioningManager#createArbitraryBucketToNode (Wenlei Xie)
• 20c7af7 Make acknowledgeResults to return future (James Sun)
• 9d47a24 Add server info client side thrift implementation (James Sun)
• 8ffd650 Abstract RemoteNodeState HTTP implementation to HttpRemoteNodeState (James Sun)
• 7416a8c Add server info server side thrift implementation (James Sun)
• 9d030b5 Update NodeState with enum values (James Sun)
• 8e44e9b Add TestHiveDistributedQueriesWithThriftRpc (James Sun)
• be031b1 Rewrite page too large exception based on network protocols (James Sun)
• 785eb79 Add thrift support for v1/task/results (James Sun)
• a2e20c7 Add thrift communication protocol config (James Sun)
• 6c9afc4 Add ThriftRpcShuffleClient implementation (James Sun)
• 2f18683 Abstract exchange HTTP client to RpcShuffleClient (James Sun)
• 5e8cae8 Rename HttpPageBufferClient to PageBufferClient (James Sun)
• f3443e7 Add thrift support for exchange server (James Sun)
• 125d77a Announce thrift port for PrestoServer (James Sun)
• 1f28b21 Initial support with Thrift RPC (Wenlei Xie)
• 4b94470 Strip netty dependency from presto-cassandra (James Sun)
• a49a812 Remove netty dependency from presto-mongodb (James Sun)
• 0c28697 Strip netty dependency from presto-elasticsearch (James Sun)
• 53e3648 Fix null pointer when using ConnectorSession (Ke Wang)
• c0fe702 Make presto-spark extensible to additional modules (Wenlei Xie)
• e64c4e2 Propagate legacyMapSubscript to selective map reader (Masha Basmanova)
• b7a74a2 Add legacyMapSubscript to SqlFunctionProperties (Masha Basmanova)
• 500c6ba Add legacy_map_subscript hidden session property (Masha Basmanova)
• dcf7f66 Add comment to ActualProperties.Global.nullsAndAnyReplicated (Ariel Weisberg)
• 9218834 Make PreferredProperties.mergeWithParent methods private (Ariel Weisberg)
• 9a68dfe Add partitioning_precision_strategy session property (Ariel Weisberg)
• 652ca6a Fix Pinot segment page source to use ColumnDataType (Devesh Agrawal)
• b6325d9 Refactor ReorderJoins to use RowExpressions (Saksham Sachdev)
• eabf46f Increase heap memory to avoid continuous Full GCs (Ajay George)
• ff6dd73 Update to Airbase 98 and Aiftlift 0.188 (Leiqing Cai)
• 32ca539 Configure DB Resource Group reload frequency (Swapnil Tailor)
• 386164e Support ALTER FUNCTION (Leiqing Cai)
• 48b2fee Add syntax support for ALTER FUNCTION (Leiqing Cai)
• 68f2d6f Format parameter types in one line in DROP FUNCTION (Leiqing Cai)
• 0278615 Use builder to construct RoutineCharacteristics (Leiqing Cai)
• 76ee53c Add end-to-end test for Presto on Spark using Docker (Andrii Rosa)
• a7aa342 Add Spark launcher (Andrii Rosa)
• f074113 Generate Presto on Spark tar.gz package (Andrii Rosa)
• cb5a00d Presto on Spark initial commit (Andrii Rosa)
• cc398f5 Add benchmark for BytesValues (Ke Wang)
• fc695e0 Optimize NOT IN filter for strings (Ke Wang)
• 422d7a9 Rename isNanAllowed function and update comment (Ke Wang)
• 479e655 Enable listing SQL functions with session property (Leiqing Cai)

[rschlussel]

Please follow the release note guidelines for the order of sections and the order of changes within a section: https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines

[bhhari]

@rschlussel can you take another pass, I had some local changes which failed to amend, which had most of the changes that you requested for. I tried to include if anything was missed. thanks

[bhhari]

@rschlussel addressed the comments

[rschlussel]

sorry, a few more comments that I missed before.

[caithagoras]

Feel free to break long lines (say > 100 or > 120 characters) for readability.