FixedSizeBinaryArray::try_from_sparse_iter failed when given all Nones

[jamescorbett]

Describe the bug

try_from_sparse_iter expects at least one item to have Some to determine the size. This is a bad assumption as it may recieve all Nones.

InvalidArgumentError("column types must match schema types, expected FixedSizeBinary(32) but found FixedSizeBinary(0) at column index 0"

To Reproduce

        let none_option: Option<[u8; 32]> = None;
        let item = arrow::array::FixedSizeBinaryArray::try_from_sparse_iter(
            vec![none_option, none_option, none_option].into_iter(),
        )
        .unwrap();
       let rb = arrow::record_batch::RecordBatch::try_new(
            arrow_schema.clone(),
            vec![std::sync::Arc::new(item)],
        )
        .unwrap();

Expected behavior

A record batch with a colum of all nulls.

Additional context

I would suggest fixing it by changing the behaviour of defaulting in array_binary.rs:560

let size = size.unwrap_or(0);

or changing the schema validation to handle a 0 size if the array is empty.

[alamb]

Thank you for the report @jamescorbett

[jackwener]

Has it already been fixed?

fn test_fixed_size_binary_array_from_sparse_iter() {
    let none_option: Option<[u8; 32]> = None;
    let input_arg = vec![none_option, none_option, none_option, none_option];
    let arr =
        FixedSizeBinaryArray::try_from_sparse_iter(input_arg.into_iter()).unwrap();
    assert_eq!(0, arr.value_length());
    assert_eq!(4, arr.len())
}

I create and run this unit test, and it is ok.

[alamb]

@jackwener one thing I noticed that that function isn't marked with #[test] (not sure if you did that locally or not).

#[test]
fn test_fixed_size_binary_array_from_sparse_iter() {
...
}

Is there any chance you can put up a PR with the test? That way we can close this ticket -- as 'already fixed' if it passes

[jamescorbett]

Has it already been fixed?

fn test_fixed_size_binary_array_from_sparse_iter() {
    let none_option: Option<[u8; 32]> = None;
    let input_arg = vec![none_option, none_option, none_option, none_option];
    let arr =
        FixedSizeBinaryArray::try_from_sparse_iter(input_arg.into_iter()).unwrap();
    assert_eq!(0, arr.value_length());
    assert_eq!(4, arr.len())
}

I create and run this unit test, and it is ok.

The issue is not this. The error occurs when using the schema.

[jackwener]

I reproduce it successfully.

Error case

    #[test]
    fn test() {
        let schema =
            Schema::new(vec![Field::new("a", DataType::FixedSizeBinary(2), false)]);
        let none_option: Option<[u8; 2]> = None;
        let item = FixedSizeBinaryArray::try_from_sparse_iter(
            vec![none_option, none_option, none_option].into_iter(),
        )
        .unwrap();
        let rb = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(item)]).unwrap();
    }

correct example

#[test]
    fn test() {
        let schema =
            Schema::new(vec![Field::new("a", DataType::FixedSizeBinary(2), false)]);
        let item = FixedSizeBinaryArray::try_from_sparse_iter(
            vec![
                None,
                Some(vec![7, 8]),
                Some(vec![9, 10]),
                None,
                Some(vec![13, 14]),
            ]
            .into_iter(),
        )
        .unwrap();
        let rb = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(item)]).unwrap();
    }

[jackwener]

The essence of the problem is that we can't get the the type of None. We need Some() to determine None type.

[jamescorbett]

The essence of the problem is that we can't get the the type None. We need Some() to determine None type.

Or the schema doesn't validate size for all none array

[jamescorbett]

Please reopen this issue. The pr does not fix the problem. See Jack's comment above for a test reproducing the issue

[alamb]

Sorry @jamescorbett -- Github got overly excited and automatically closed the PR

[alamb]

Here is a PR to add the test: #1551