Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve parquet WriterProperites and ReaderProperties docs #4392

Merged
merged 1 commit into from
Jun 9, 2023
Merged

Improve parquet WriterProperites and ReaderProperties docs #4392

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 74 additions & 65 deletions parquet/src/file/properties.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,55 +15,7 @@
// specific language governing permissions and limitations
// under the License.

//! [`WriterProperties`]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved the examples to the WriterProperties and ReaderProperties so they would be easier to find

I think the module level docs now look pretty clear:

Screenshot 2023-06-09 at 11 07 30 AM

//!
//! # Usage
//!
//! ```rust
//! use parquet::{
//! basic::{Compression, Encoding},
//! file::properties::*,
//! schema::types::ColumnPath,
//! };
//!
//! // Create properties with default configuration.
//! let props = WriterProperties::default();
//!
//! // Use properties builder to set certain options and assemble the configuration.
//! let props = WriterProperties::builder()
//! .set_writer_version(WriterVersion::PARQUET_1_0)
//! .set_encoding(Encoding::PLAIN)
//! .set_column_encoding(ColumnPath::from("col1"), Encoding::DELTA_BINARY_PACKED)
//! .set_compression(Compression::SNAPPY)
//! .build();
//!
//! assert_eq!(props.writer_version(), WriterVersion::PARQUET_1_0);
//! assert_eq!(
//! props.encoding(&ColumnPath::from("col1")),
//! Some(Encoding::DELTA_BINARY_PACKED)
//! );
//! assert_eq!(
//! props.encoding(&ColumnPath::from("col2")),
//! Some(Encoding::PLAIN)
//! );
//! ```
//!
//! Reader properties.
//!
//! # Usage
//!
//! ```rust
//! use parquet::file::properties::ReaderProperties;
//!
//! // Create properties with default configuration.
//! let props = ReaderProperties::builder().build();
//!
//! // Use properties builder to set certain options and assemble the configuration.
//! let props = ReaderProperties::builder()
//! .set_backward_compatible_lz4(false)
//! .build();
//! ```

//! Configuration via [`WriterProperties`] and [`ReaderProperties`]
use std::{collections::HashMap, sync::Arc};

use crate::basic::{Compression, Encoding};
Expand All @@ -72,20 +24,30 @@ use crate::file::metadata::KeyValue;
use crate::format::SortingColumn;
use crate::schema::types::ColumnPath;

const DEFAULT_PAGE_SIZE: usize = 1024 * 1024;
const DEFAULT_WRITE_BATCH_SIZE: usize = 1024;
const DEFAULT_WRITER_VERSION: WriterVersion = WriterVersion::PARQUET_1_0;
const DEFAULT_COMPRESSION: Compression = Compression::UNCOMPRESSED;
const DEFAULT_DICTIONARY_ENABLED: bool = true;
const DEFAULT_DICTIONARY_PAGE_SIZE_LIMIT: usize = DEFAULT_PAGE_SIZE;
const DEFAULT_STATISTICS_ENABLED: EnabledStatistics = EnabledStatistics::Page;
const DEFAULT_MAX_STATISTICS_SIZE: usize = 4096;
const DEFAULT_MAX_ROW_GROUP_SIZE: usize = 1024 * 1024;
const DEFAULT_CREATED_BY: &str =
/// Default value for [`WriterProperties::data_page_size_limit`]
pub const DEFAULT_PAGE_SIZE: usize = 1024 * 1024;
/// Default value for [`WriterProperties::write_batch_size`]
pub const DEFAULT_WRITE_BATCH_SIZE: usize = 1024;
/// Default value for [`WriterProperties::writer_version`]
pub const DEFAULT_WRITER_VERSION: WriterVersion = WriterVersion::PARQUET_1_0;
/// Default value for [`WriterProperties::compression`]
pub const DEFAULT_COMPRESSION: Compression = Compression::UNCOMPRESSED;
/// Default value for [`WriterProperties::dictionary_enabled`]
pub const DEFAULT_DICTIONARY_ENABLED: bool = true;
/// Default value for [`WriterProperties::dictionary_page_size_limit`]
pub const DEFAULT_DICTIONARY_PAGE_SIZE_LIMIT: usize = DEFAULT_PAGE_SIZE;
/// Default value for [`WriterProperties::statistics_enabled`]
pub const DEFAULT_STATISTICS_ENABLED: EnabledStatistics = EnabledStatistics::Page;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I was looking for, but since the value was not pub it wasn't in the rustdocs

I could have included the value in the text of WriterProperties::statistics_enabled but thought that this way was less likely to get out of sync

/// Default value for [`WriterProperties::max_statistics_size`]
pub const DEFAULT_MAX_STATISTICS_SIZE: usize = 4096;
/// Default value for [`WriterProperties::max_row_group_size`]
pub const DEFAULT_MAX_ROW_GROUP_SIZE: usize = 1024 * 1024;
/// Default value for [`WriterProperties::created_by`]
pub const DEFAULT_CREATED_BY: &str =
concat!("parquet-rs version ", env!("CARGO_PKG_VERSION"));
/// default value for the false positive probability used in a bloom filter.
/// Default value for [`BloomFilterProperties::fpp`]
pub const DEFAULT_BLOOM_FILTER_FPP: f64 = 0.05;
/// default value for the expected number of distinct values used in a bloom filter.
/// Default value for [`BloomFilterProperties::ndv`]
pub const DEFAULT_BLOOM_FILTER_NDV: u64 = 1_000_000_u64;

/// Parquet writer version.
Expand All @@ -111,10 +73,41 @@ impl WriterVersion {
/// Reference counted writer properties.
pub type WriterPropertiesPtr = Arc<WriterProperties>;

/// Writer properties.
/// Configuration settings for writing parquet files.
///
/// All properties except the key-value metadata are immutable,
/// use [`WriterPropertiesBuilder`] to assemble these properties.
///
/// # Example
///
/// ```rust
/// use parquet::{
/// basic::{Compression, Encoding},
/// file::properties::*,
/// schema::types::ColumnPath,
/// };
///
/// // Create properties with default configuration.
/// let props = WriterProperties::default();
///
/// // Use properties builder to set certain options and assemble the configuration.
/// let props = WriterProperties::builder()
/// .set_writer_version(WriterVersion::PARQUET_1_0)
/// .set_encoding(Encoding::PLAIN)
/// .set_column_encoding(ColumnPath::from("col1"), Encoding::DELTA_BINARY_PACKED)
/// .set_compression(Compression::SNAPPY)
/// .build();
///
/// assert_eq!(props.writer_version(), WriterVersion::PARQUET_1_0);
/// assert_eq!(
/// props.encoding(&ColumnPath::from("col1")),
/// Some(Encoding::DELTA_BINARY_PACKED)
/// );
/// assert_eq!(
/// props.encoding(&ColumnPath::from("col2")),
/// Some(Encoding::PLAIN)
/// );
/// ```
#[derive(Debug, Clone)]
pub struct WriterProperties {
data_page_size_limit: usize,
Expand Down Expand Up @@ -307,7 +300,8 @@ impl WriterProperties {
}
}

/// Writer properties builder.
/// Builder for parquet file writer configuration. See example on
/// [`WriterProperties`]
pub struct WriterPropertiesBuilder {
data_page_size_limit: usize,
dictionary_page_size_limit: usize,
Expand Down Expand Up @@ -809,10 +803,24 @@ pub type ReaderPropertiesPtr = Arc<ReaderProperties>;

const DEFAULT_READ_BLOOM_FILTER: bool = false;

/// Reader properties.
/// Configuration settings for reading parquet files.
///
/// All properties are immutable and `Send` + `Sync`.
/// Use [`ReaderPropertiesBuilder`] to assemble these properties.
///
/// # Example
///
/// ```rust
/// use parquet::file::properties::ReaderProperties;
///
/// // Create properties with default configuration.
/// let props = ReaderProperties::builder().build();
///
/// // Use properties builder to set certain options and assemble the configuration.
/// let props = ReaderProperties::builder()
/// .set_backward_compatible_lz4(false)
/// .build();
/// ```
pub struct ReaderProperties {
codec_options: CodecOptions,
read_bloom_filter: bool,
Expand All @@ -835,7 +843,8 @@ impl ReaderProperties {
}
}

/// Reader properties builder.
/// Builder for parquet file reader configuration. See example on
/// [`ReaderProperties`]
pub struct ReaderPropertiesBuilder {
codec_options_builder: CodecOptionsBuilder,
read_bloom_filter: Option<bool>,
Expand Down