Nullable field with nested not nullable map in json

[spebern]

Which part is this question about

New raw json decoder.

Describe your question

While trying to use the new json decoder in delta I stumbled over the following issue:

#[test]
fn test_delta_checkpoint() {
    let json = "{\"protocol\":{\"minReaderVersion\":1,\"minWriterVersion\":2}}";
    let schema = Arc::new(Schema::new(vec![
        Field::new(
            "protocol",
            DataType::Struct(vec![
                Field::new("minReaderVersion", DataType::Int32, true),
                Field::new("minWriterVersion", DataType::Int32, true),
            ]),
            true,
        ),
        Field::new(
            "add",
            DataType::Struct(vec![Field::new(
                "partitionValues",
                DataType::Map(
                    Box::new(Field::new(
                        "key_value",
                        DataType::Struct(vec![
                            Field::new("key", DataType::Utf8, false),
                            Field::new("value", DataType::Utf8, true),
                        ]),
                        false,
                    )),
                    false,
                ),
                false, // <-- when this is true the test passes
            )]),
            true,
        ),
    ]));

    let batches = do_read(json, 1024, true, schema);
    assert_eq!(batches.len(), 1);
}

This fails with:

thread 'raw::tests::test_delta_checkpoint' panicked at 'called `Result::unwrap()` on an `Err` value: JsonError("expected { got null")', arrow-json/src/raw/mod.rs:389:18

It is related to nested map being not nullable (see comment in the code snippet). I'd expect that if a field (in the example "add") is null, the inner "not nullables" should
not get checked.

Does this work as expected or could this be a bug?

[tustvold]

This is a bug, and I understand what causes it, thank you for the report. Working on a fix