Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Table features interfaces/utilities for Kernel #4

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,55 @@ public class TableConfig<T> {
// TableConfigs //
//////////////////

/**
* Whether this Delta table is append-only. Files can't be deleted, or values can't be updated.
*/
public static final TableConfig<Boolean> APPEND_ONLY =
new TableConfig<>(
"delta.appendOnly",
"false",
Boolean::valueOf,
value -> true,
"needs to be a boolean.",
true);

/**
* Enable change data feed output. When enabled, DELETE, UPDATE, and MERGE INTO operations will
* need to do additional work to output their change data in an efficiently readable format.
*/
public static final TableConfig<Boolean> CHANGE_DATA_FEED =
new TableConfig<>(
"delta.enableChangeDataFeed",
"false",
Boolean::valueOf,
value -> true,
"needs to be a boolean.",
true);

/** Whether commands modifying this Delta table are allowed to create new deletion vectors. */
public static final TableConfig<Boolean> ENABLE_DELETION_VECTORS_CREATION =

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would it be nice if all of our table-feature-enablement-related TableConfigs ended with _ENABLED?

e.g. APPEND_ONLY_ENABLED, CHANGE_DATA_FEED_ENABLED, DELETION_VECTORS_CREATION_ENBABLED

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated.

new TableConfig<>(
"delta.enableDeletionVectors",
"false",
Boolean::valueOf,
value -> true,
"needs to be a boolean.",
true);

/**
* Indicates whether Row Tracking is enabled on the table. When this flag is turned on, all rows
* are guaranteed to have Row IDs and Row Commit Versions assigned to them, and writers are
* expected to preserve them by materializing them to hidden columns in the data files.
*/
public static final TableConfig<Boolean> ROW_TRACKING_ENABLED =
new TableConfig<>(
"delta.enableRowTracking",
"false",
Boolean::valueOf,
value -> true,
"needs to be a boolean.",
true);

/**
* The shortest duration we have to keep logically deleted data files around before deleting them
* physically.
Expand Down Expand Up @@ -170,6 +219,19 @@ public class TableConfig<T> {
"needs to be a boolean.",
true);

/**
* Whether widening the type of an existing column or field is allowed, either manually using
* ALTER TABLE CHANGE COLUMN or automatically if automatic schema evolution is enabled.
*/
public static final TableConfig<Boolean> ENABLE_TYPE_WIDENING =
new TableConfig<>(
"delta.enableTypeWidening",
"false",
Boolean::valueOf,
value -> true,
"needs to be a boolean.",
true);

/** All the valid properties that can be set on the table. */
private static final Map<String, TableConfig<?>> VALID_PROPERTIES =
Collections.unmodifiableMap(
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
import io.delta.kernel.internal.metrics.SnapshotQueryContext;
import io.delta.kernel.internal.metrics.SnapshotReportImpl;
import io.delta.kernel.internal.snapshot.SnapshotManager;
import io.delta.kernel.internal.tablefeatures.TableFeatures;
import io.delta.kernel.internal.util.Clock;
import io.delta.kernel.metrics.SnapshotReport;
import io.delta.kernel.types.StructField;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import io.delta.kernel.internal.replay.LogReplay;
import io.delta.kernel.internal.snapshot.LogSegment;
import io.delta.kernel.internal.snapshot.SnapshotHint;
import io.delta.kernel.internal.tablefeatures.TableFeatures;
import io.delta.kernel.internal.util.ColumnMapping;
import io.delta.kernel.internal.util.ColumnMapping.ColumnMappingMode;
import io.delta.kernel.internal.util.SchemaUtils;
Expand Down Expand Up @@ -154,9 +155,9 @@ public Transaction build(Engine engine) {
if (!newWriterFeatures.isEmpty()) {
logger.info("Automatically enabling writer features: {}", newWriterFeatures);
shouldUpdateProtocol = true;
List<String> oldWriterFeatures = protocol.getWriterFeatures();
Set<String> oldWriterFeatures = protocol.getWriterFeatures();
protocol = protocol.withNewWriterFeatures(newWriterFeatures);
List<String> curWriterFeatures = protocol.getWriterFeatures();
Set<String> curWriterFeatures = protocol.getWriterFeatures();
checkArgument(!Objects.equals(oldWriterFeatures, curWriterFeatures));
TableFeatures.validateWriteSupportedTable(
protocol, metadata, metadata.getSchema(), table.getPath(engine));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
import io.delta.kernel.internal.replay.ConflictChecker;
import io.delta.kernel.internal.replay.ConflictChecker.TransactionRebaseState;
import io.delta.kernel.internal.rowtracking.RowTracking;
import io.delta.kernel.internal.tablefeatures.TableFeatures;
import io.delta.kernel.internal.util.*;
import io.delta.kernel.metrics.TransactionReport;
import io.delta.kernel.types.StructType;
Expand Down
Loading
Loading