[Iceberg] Write NDVs to Iceberg table format

[ZacBlanco]

Description

This PR introduces the changes required to read+write the distinct value count statistics as described by Iceberg's Puffin file specification ¹.

The change can be broken down into three main parts.

1. Updates to the SPI to allow connectors to define the function used to calculate a specific statistic.
2. The addition of 3 new functions: sketch_theta, sketch_theta_estimate, and sketch_theta_summary.
3. Plumbing and implementation in the Iceberg connector to support reading+writing of the NDVs

SPI Changes

The SPI changes are minimal, but it involves adding a new field to the ColumnStatisticMetadata class in order to
allow connectors to override the functions used during ANALYZE. Previously, the functions for each ColumnStatisticType were defined statically in a switch block inside of presto-main. The logic in the switch block
checked the statistic type and then mapped that to a function which executed.

Instead, this PR introduces a change where each statistic has a default function of which the name is associated
in the ColumnStatisticType enum. However, the change here adds a field in the ColumnStatisticMetadata for a
function name. For most connectors that implement analyze, they can easily generate a ColumnStatisticMetadata
instance using the default function, however, we will now also allow a connector to create a ColumnStatisticMetadata
instance which has a custom function name that can be defined by the connector. That function is then eventually
resolved by the coordinator and executed during the ANALYZE, and the result of the function is returned to the
connector in the finishStatisticsCollection function as a Block type which can be read by connector-defined
logic. This design keeps all of the logic for defining and parsing the result up to the connector, so no additional changes
need to be made to the ANALYZE design other than a minor refactor.

New Sketch Functions

Iceberg's Puffin spec defines the format that NDVs must be written in. Currently, the only available format is a binary
blob representing an Apache Datasketches Theta Sketch ². I've added Apache Datasketches as a presto dependency
and implemented three basic functions which expose the sketch so that Iceberg can eventually consume it when writing
statistics.

A brief note on naming: I presume in the future we will need support for KLL sketches from the same library, as the latest Hive 4.0 beta release defines its histograms format as a KLL sketch, so to make it easier to find the "sketch" families of function when sorted alphabetically, I've decided to use the naming convention sketch_*

1. sketch_theta(<column>) -> varbinary: An aggregation function which accepts a column and generates a binary representation of the org.apache.datasketches.theta.CompactSketch. Applications can easily consume this raw binary
   format to gain access to a CompactSketch instance and associated methods.
2. sketch_theta_estimate(<varbinary sketch>) -> double: A scalar function which consumes a raw binary sketch and produces the estimate. This is effectively the same as calling CompactSketch::getEstimate. I've exposed this as a convenience for checking the sketch's output
3. sketch_theta_summary(<varbinary sketch>) -> row(estimate double, theta double, upper_bound_std1 double, lower_bound_std1 double, retained_entries int): This is another scalar function, but returns a row type containing
   more human-readable information about the sketch such as the theta parameter as well as upper and lower bounds
   for 1 standard deviation from the estimate

Iceberg Read/Write Support

Lastly, this change includes the additional plumbing required to read and write NDVs within the Iceberg connector's
TableStatisticsMaker class.

On the statistics write path inside of the IcebergAbstractMetadata::finishStatisticsCollection
We add a call to writeTableStatistics. This will take all of the collected statistics data and write out the theta
sketches collected into blobs within a puffin file, then the table metadata is updated with a pointer to the puffin file.

On the read path, we now check for the most up-to-date statistics file, pick that file, and read all of the blobs' metadata.
For each blob we check the "ndv" property (see the puffin spec¹) and associated fieldId, and add it to a map of field -> ColumnStatistics.
The code is structured in a way that should make it easy to read/look for more statistics in the file as they are added
to the Puffin spec. After reading the statistic file data, the resulting map of column statistics is used as a "base" which
the additional existing statistics can then be added to, such as the min, max, etc.

Motivation and Context

In order for Presto's optimizer to do its job more effectively, statistics are useful to determine the resulting size of joins or predicate filters. By having them available, the optimizer can make more informed decisions.

Impact

• 3 new functions available in the Presto default function namespace
• ANALYZE available to the Iceberg regardless of metastore

Test Plan

• existing tests should cover SPI changes.
• Tests added for new sketch functions
• Tests added for ANALYZE and getTableStatistics

Contributor checklist

• Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

== RELEASE NOTES ==

General Changes
* The Iceberg connector will now read+write distinct value counts as defined by the spec.
* Functions to compute Theta sketches from the Apache DataSketches library are now available

Footnotes

1. https://iceberg.apache.org/puffin-spec/ ↩ ↩²

2. https://datasketches.apache.org/docs/Theta/ThetaSketchFramework.html ↩

[ClarenceThreepwood]

Overall looks good. Mostly nits + more tests would be good

[ClarenceThreepwood]

I think you may need to override isCalledOnNullInput

[ZacBlanco]

The default of isCalledOnNullInput is false, but I put this here as a defensive measure. I wasn't aware of any contract in the calling convention. I guess if isCalledOnNullInput takes care of it then I can probably remove it.

[ClarenceThreepwood]

The contract for isCalledOnNullInput is described here. It doesn't change the behavior of the function, but is instead a directive that is used by the optimizer for certain rewrites

presto> select sketch_theta(cast(null as varbinary));
_col0

---

01 03 03 00 00 1e 00 00
(1 row)

presto> select sketch_theta(null);
_col0

---

01 03 03 00 00 1e 00 00
(1 row)

[hantangwangd]

Overall looks good to me. Just some small nits.

[aaneja]

Still need to review the tests; have taken a look on the wireup E2E and the theta sketch functions

[hantangwangd]

Looks good to me. Only one small problem in test case.

[ZacBlanco]

@steveburnett can you take a look at the additional docs I added in f83256c54dd811827fb30226b23f95eb585a705c? Thanks!

[github-actions]

Codenotify: Notifying subscribers in CODENOTIFY files for diff 541d7ec...3f5ee7f.

Notify	File(s)
@steveburnett	presto-docs/src/main/sphinx/functions.rst presto-docs/src/main/sphinx/functions/sketch.rst

[hantangwangd]

LGTM! Thanks for the job, it's a very useful feature.

[steveburnett]

Great job on the docs! Only one small nit.

[steveburnett]

LGTM! (docs)

New local build shows the updated doc. Thanks!

[liushengxuan]

@beinan Thanks for inviting!!

LGTM. Thanks for the contribution!!