Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-459: Add Variant logical type annotation #460

Merged
merged 5 commits into from
Nov 6, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions LogicalTypes.md
Original file line number Diff line number Diff line change
Expand Up @@ -563,6 +563,23 @@ defined by the [BSON specification][bson-spec].

The sort order used for `BSON` is unsigned byte-wise comparison.

### VARIANT

`VARIANT` is used for a Variant value. It must annotate a group. The group must
contain a `binary` field named `metadata`, and a `binary` field named `value`.
rdblue marked this conversation as resolved.
Show resolved Hide resolved
The `VARIANT` annotated group can be used to store either an unshredded Variant
value, or a shredded Variant value.

* The top level must be a group annotated with `VARIANT` that contains a
`binary` field named `metadata`, and a `binary` field named `value`.
rdblue marked this conversation as resolved.
Show resolved Hide resolved
* Additional fields which start with `_` (underscore) can be ignored.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed? None of the other types allow writing columns that should be ignored.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was desired in case there were some additional (but redundant) metadata or values we might store, and still allow it to be a valid Variant value (group).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that we want to add ignored columns. If we need to update the spec because something is missing, we should just do that directly instead of working around it with unspecified columns that only work in certain proprietary cases.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I was worried that future evolution could break existing stored Variants, but simply adding a new field with optional or redundant semantics achieves the same compatibility story. This is removed.

* If `metadata` and `value` are the only fields in the group, then the group
is an unshredded Variant value. The `metadata` and `value` fields are
interpreted as an encoded Variant value as defined by the
[Variant binary encoding specification](VariantEncoding.md).
* If the group contains additional fields, it is a shredded Variant, and must
adhere to the scheme detailed in the [Variant shredding specification](VariantShredding.md).
rdblue marked this conversation as resolved.
Show resolved Hide resolved

## Nested Types

This section specifies how `LIST` and `MAP` can be used to encode nested types
Expand Down
2 changes: 1 addition & 1 deletion VariantEncoding.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ This document describes the Variant Binary Encoding scheme.
[VariantShredding.md](VariantShredding.md) describes the details of the Variant shredding scheme.

# Variant in Parquet
A Variant value in Parquet is represented by a group with 2 fields, named `value` and `metadata`.
A Variant value in Parquet is represented by a group annotated with `VARIANT`, with 2 fields, named `value` and `metadata`.
rdblue marked this conversation as resolved.
Show resolved Hide resolved
Both fields `value` and `metadata` are of type `binary`, and cannot be `null`.

# Metadata encoding
Expand Down
Loading