Add support for Structs evolution in MapType keys and values

[johanl-db]

Description

This PR resolves issue #1641 to allow automatic schema evolution in structs that are inside maps.

Assuming the target and source tables have the following schemas:
target: id string, map map<int, struct<a: int, b: int>>
source: id string, map map<int, struct<a: int, b: int, c: int>>

SET spark.databricks.delta.schema.autoMerge.enabled = true;

MERGE INTO target t
USING source s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET *

returns an analysis error today:

AnalysisException: cannot resolve 's.map' due to data type mismatch: cannot cast map<string,struct<a:int,b:int>> to map<string,struct<a:int,b:int,c:string>>;

With this change, the merge command succeeds and the target table schema evolves to include field c inside the map value. The same also works for map keys.

How was this patch tested?

• Tests are added to MergeIntoSuiteBase and MergeIntoSQLSuite to cover struct evolution inside of maps values and keys.

Does this PR introduce any user-facing changes?

Yes, struct evolution inside of maps now succeeds instead of failing with an analysis error, see previous example.

[fred-db]

Nice fix! I had some minor comments. From what I can tell, this also fixes schema evolution in arrays nested in maps. Could we have one additional test for this for completeness?

[fred-db]

nit: merge onto one line
withTable("source", "target") {

[johanl-db]

Both tests in MergeIntoSQLSuite are removed after I also implemented struct evolution inside of map keys and not just in map values. There actually was not good reason to limit it to map values only.

[johanl-db]

Nice fix! I had some minor comments. From what I can tell, this also fixes schema evolution in arrays nested in maps. Could we have one additional test for this for completeness?

I added a number of tests to cover map keys evolution and two tests to cover arrays nested in maps.

[fred-db]

Could we use the testNestedStructsEvolution helper method? It makes the tests super readable

[johanl-db]

Unfortunately, testNestedStructsEvolution relies on parsing JSON data which is what makes it more readable but JSON requires the keys in maps to be strings which is too limiting for these tests.

[fred-db]

LGTM!