Make max slice size in ORC slice reader configurable

[sdruzkin]

Description

ORC slice reader has a hardcoded max slice size of 1GB, it throws when it one attempts to read a slice(s) larger than 1GB. Make it configurable to be able to increase the threshold in Spark for some failing jobs.

Plumbed the new value trough OrcReaderOptions -> OrcRecordReaderOptions -> OrcReader -> SelectiveReaderContext -> SliceDirectSelectiveStreamReader.

Motivation and Context

Hardcoded value of 1GB is too low for some files and needs to be increased to accommodate such cases.

Impact

No impact.

Test Plan

Existing and new unit tests.

Contributor checklist

• Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== NO RELEASE NOTE ==

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D66800897

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D66800897

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D66800897

[rschlussel]

why do we have this limit in the first place? Is the problem that we aren't reserving the memory for the slice before we read?

[steveburnett]

If this is a new configuration (or session) property, please add documentation in the appropriate pages of the Presto doc:

Presto Session Properties
Presto Configuration Properties
Presto C++ Session Properties
Presto C++ Configuration Properties

[sdruzkin]

why do we have this limit in the first place? Is the problem that we aren't reserving the memory for the slice before we read?

I guess so, ORC memory context does not do any good with memory reservation. It was added somewhere before 2021-2022, when a typical cluster had under 30GB of total memory and a very little headroom.

If this is a new configuration (or session) property, please add documentation in the appropriate pages of the Presto doc:

This setting cannot be configured through the session or cluster properties.