Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-48759][SQL] Add migration doc for CREATE TABLE AS SELECT behav…
…ior change behavior change since Spark 3.4 ### What changes were proposed in this pull request? Add migration guide for `CREATE TABLE AS SELECT...` behavior change. SPARK-41859 changes the behaviour for `CREATE TABLE AS SELECT ...` from OVERWRITE to APPEND when `spark.sql.legacy.allowNonEmptyLocationInCTAS` is set to `true`: ``` drop table if exists test_table; create table test_table location '/tmp/test_table' stored as parquet as select 1 as col union all select 2 as col; drop table if exists test_table; create table test_table location '/tmp/test_table' stored as parquet as select 3 as col union all select 4 as col; select * from test_table; ``` This produces {3, 4} in Spark <3.4.0 and {1, 2, 3, 4} in Spark 3.4.0 and later. This is a silent change in `spark.sql.legacy.allowNonEmptyLocationInCTAS` behaviour which introduces wrong results in the user application. ### Why are the changes needed? This documents a behavior change starting in Spark 3.4 for `CREATE TABLE AS SELECT` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? `doc build ` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47152 from asl3/allowNonEmptyLocationInCTAS. Authored-by: Amanda Liu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
- Loading branch information