[Improvement] Clarify the definitions and behaviors of interfaces dropTable and purgeTable

[mchades]

What would you like to be improved?

The described of dropTable is:

Drop a table from the catalog.

and the purgeTable is:

Drop a table from the catalog and completely remove its data.
If the catalog supports to purge a table, this method should be overridden. The default implementation throws an UnsupportedOperationException.

However, the original intention behind designing these two interfaces was that:

• dropTable should delete the metadata and data of the table simultaneously. (If a recycle bin is available, the deleted data should go to the recycle bin.)
• purgeTable is intended for catalogs with a recycle bin feature, ensuring that the table data is deleted from the recycle bin after dropTable. Otherwise, an UnsupportedOperationException should be thrown.

How should we improve?

Improving the method comment, checking the method call and implementation, and making sure the method's invocation scenario is in line with the semantic meaning of the method.

• [Subtask] Ensure the Hive catalog implements the correct semantics of dropTable and purgeTable #1439
• [Subtask] Ensure the Iceberg catalog implements the correct semantics of dropTable and purgeTable #1441
• [Subtask] Ensure the JDBC catalogs implement the correct semantics of dropTable and purgeTable #1442
• [Subtask] Ensure the Trino connector is calling the correct methods of dropTable and purgeTable #1443

[mchades]

@jerryshao @yuqi1129 @diqiu50 @FANNG1 @Clearvive @qqqttt123 Do you have any suggestions on this?

[jerryshao]

LGTM, let's check it separately.

[FANNG1]

Iceberg has different meanings, dropTable is drop metadata while purge table is drop metadata and data.
@qqqttt123 , do you know any background?

[mchades]

Iceberg has different meanings, dropTable is drop metadata while purge table is drop metadata and data. @qqqttt123 , do you know any background?

Can you provide some reference for that? @FANNG1

[diqiu50]

After that, should the DROP TABLE statement in Trino call the drop table API instead of using purgeTable?

[FANNG1]

Iceberg has different meanings, dropTable is drop metadata while purge table is drop metadata and data. @qqqttt123 , do you know any background?

Can you provide some reference for that? @FANNG1

see https://iceberg.apache.org/docs/latest/spark-ddl/#drop-table-purge

and dropTable in catalog interface

  /**
   * Drop a table; optionally delete data and metadata files.
   *
   * <p>If purge is set to true the implementation should delete all data and metadata files.
   *
   * @param identifier a table identifier
   * @param purge if true, delete all data and metadata files in the table
   * @return true if the table was dropped, false if the table did not exist
   */
  boolean dropTable(TableIdentifier identifier, boolean purge);

[qqqttt123]

Iceberg has different meanings, dropTable is drop metadata while purge table is drop metadata and data. @qqqttt123 , do you know any background?

I don't think they have different meaning. Iceberg is based on object storage. Object storage may not have their own trash mechnism. Maybe we can treat the snapshot of Iceberg as a trash. So DropTable don't delete data. They can be garbaged.

[mchades]

After that, should the DROP TABLE statement in Trino call the drop table API instead of using purgeTable?

I think in most scenarios it is, depending on the semantics of Trino's Drop TABLE statement. @diqiu50

[FANNG1]

Iceberg has different meanings, dropTable is drop metadata while purge table is drop metadata and data. @qqqttt123 , do you know any background?

I don't think they have different meaning. Iceberg is based on object storage. Object storage may not have their own trash mechnism. Maybe we can treat the snapshot of Iceberg as a trash. So DropTable don't delete data. They can be garbaged.

yes, agree.