Skip to content

Commit

Permalink
[Doc] Type Widening documentation (#3025)
Browse files Browse the repository at this point in the history
## Description
Add a documentation page for the type widening table feature, in preview
in Delta. 3.2

## How was this patch tested?
N/A
  • Loading branch information
johanl-db authored May 6, 2024
1 parent 1ba4832 commit 8cb2e78
Show file tree
Hide file tree
Showing 4 changed files with 83 additions and 0 deletions.
1 change: 1 addition & 0 deletions docs/source/delta-drop-feature.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ To remove a Delta table feature, you run an `ALTER TABLE <table-name> DROP FEATU
You can drop the following Delta table features:

- `deletionVectors`. See [_](delta-deletion-vectors.md).
- `typeWidening-preview`. See [_](delta-type-widening.md). Type widening is available in preview in <Delta> 3.2.0 and above.
- `v2Checkpoint`. See [V2 Checkpoint Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec). Drop support for V2 Checkpoints is available in <Delta> 3.1.0 and above.

You cannot drop other [Delta table features](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#valid-feature-names-in-table-features).
Expand Down
79 changes: 79 additions & 0 deletions docs/source/delta-type-widening.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
description: Learn about type widening in Delta.
---

# Delta type widening

.. note:: This feature is available in preview in <Delta> 3.2.

The type widening feature allows changing the type of columns in a Delta table to a wider type. This enables manual type changes using the `ALTER TABLE ALTER COLUMN` command and automatic type migration with schema evolution in `INSERT` and `MERGE INTO` commands.

## Supported type changes

The feature preview in <Delta> 3.2 supports a limited set of type changes:
- `BYTE` to `SHORT` and `INT`.
- `SHORT` to `INT`

Type changes are supported for top-level columns as well as fields nested inside structs, maps and arrays.

## How to enable <Delta> type widening

.. important::

Enabling type widening sets the Delta table feature `typeWidening-preview`, a reader/writer protocol feature. Only clients that support this table feature can read and write to the table once the table feature is set. You must use <Delta> 3.2 or above to read and write to such Delta tables.

You can enable type widening on an existing table by setting the `delta.enableTypeWidening` table property to `true`:

```sql
ALTER TABLE <table_name> SET TBLPROPERTIES ('delta.enableTypeWidening' = 'true')
```

Alternatively, you can enable type widening during table creation:

```sql
CREATE TABLE T(c1 INT) USING DELTA TBLPROPERTIES('delta.enableTypeWidening' = 'true')
```

To disable type widening:

```sql
ALTER TABLE <table_name> SET TBLPROPERTIES ('delta.enableTypeWidening' = 'false')
```

Disabling type widening prevents future type changes from being applied to the table. It doesn't affect type changes previously applied and in particular, it doesn't remove the type widening table feature and doesn't allow clients that don't support the type widening table feature to read and write to the table.

To remove the type widening table feature from the table and allow other clients that don't support this feature to read and write to the table, see [_](#removing-the-type-widening-table-feature).

## Manually applying a type change

When type widening is enabled on a Delta table, you can change the type of a column using the `ALTER COLUMN` command:

```sql
ALTER TABLE <table_name> ALTER COLUMN <col_name> TYPE <new_type>
```

The table schema is updated without rewriting the underlying Parquet files.

## Type changes with automatic schema evolution
Type changes are applied automatically during ingestion with `INSERT` and `MERGE INTO` commands when:
- Type widening is enabled on the target table.
- The command runs with [automatic schema evolution](delta-update.md#merge-schema-evolution) enabled.
- The source column type is wider than the target column type.
- Changing the target column type to the source type is a [supported type change](#supported-type-changes)

When all conditions are satisfied, the target table schema is updated automatically to change the target column type to the source column type.

## Removing the type widening table feature

The type widening feature can be removed from a Delta table using the `DROP FEATURE` command:

```sql
ALTER TABLE <table-name> DROP FEATURE 'typeWidening-preview' [TRUNCATE HISTORY]
```

See [_](delta-drop-feature.md) for more information on dropping Delta table features.

When dropping the type widening feature, the underlying Parquet files are rewritten when necessary to ensure that the column types in the files match the column types in the Delta table schema.
After the type widening feature is removed from the table, Delta clients that don't support the feature can read and write to the table.

.. include:: /shared/replacements.md
1 change: 1 addition & 0 deletions docs/source/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ This is the documentation site for <Delta>.
delta-drop-feature
delta-apidoc
delta-storage
delta-type-widening
delta-uniform
delta-sharing
concurrency-control
Expand Down
2 changes: 2 additions & 0 deletions docs/source/versioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ The following <Delta> features break forward compatibility. Features are enabled
V2 Checkpoints, [Delta Lake 3.0.0](https://github.com/delta-io/delta/releases/tag/v3.0.0),[V2 Checkpoint Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec)
Domain metadata, [Delta Lake 3.0.0](https://github.com/delta-io/delta/releases/tag/v3.0.0),[Domain Metadata Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#domain-metadata)
Clustering, [Delta Lake 3.1.0](https://github.com/delta-io/delta/releases/tag/v3.1.0),[_](/delta-clustering.md)
Type widening (Preview),[Delta Lake 3.2.0](https://github.com/delta-io/delta/releases/tag/v3.2.0),[_](/delta-type-widening.md)

<a id="table-protocol"></a>

Expand Down Expand Up @@ -106,6 +107,7 @@ The following table shows minimum protocol versions required for <Delta> feature
Timestamp without Timezone,7,3,[TimestampNTZType](https://spark.apache.org/docs/latest/sql-ref-datatypes.html)
Iceberg Compatibility V1,7,2,[IcebergCompatV1](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#iceberg-compatibility-v1)
V2 Checkpoints,7,3,[V2 Checkpoint Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec)
Type widening (Preview),7,3,[_](/delta-type-widening.md)

<a id="upgrade"></a>

Expand Down

0 comments on commit 8cb2e78

Please sign in to comment.