A trait for append_value and append_null on ArrayBuilders

[dnsco]

I'm generating an arrow schema, and because there is not a macro, when it's time set the values I need to invoke a macro like this:

macro_rules! set_value {
    ($builder:expr,$i:expr,$typ:ty,$getter:ident,$value:expr) => {{
        let b: &mut $typ = $builder.field_builder::<$typ>($i).unwrap();

        match $value {
            Some(cow) => {
                let v = cow.$getter().unwrap();
                b.append_value(v)
            }
            None => b.append_null(),
        }
    }};
}

fn append_value(
    f: &Field,
    builder: &mut StructBuilder,
    i: usize,
    value: Option<Cow<Value>>,
) -> Result<()> {
    match f.data_type() {
        DataType::Float64 => set_value!(builder, i, Float64Builder, as_f64, value),
        DataType::Float32 => set_value!(builder, i, Float32Builder, as_f32, value),
        DataType::Int64 => set_value!(builder, i, Int64Builder, as_i64, value),
        DataType::Int32 => set_value!(builder, i, Int32Builder, as_i32, value),
        DataType::UInt64 => set_value!(builder, i, UInt64Builder, as_u64, value),
        DataType::UInt32 => set_value!(builder, i, UInt32Builder, as_u32, value),
        DataType::Utf8 => set_value!(builder, i, StringBuilder, as_str, value),
        DataType::LargeUtf8 => set_value!(builder, i, LargeStringBuilder, as_str, value),
        DataType::Binary => set_value!(builder, i, BinaryBuilder, as_bytes, value),
        DataType::LargeBinary => set_value!(builder, i, LargeBinaryBuilder, as_bytes, value),
        DataType::Boolean => set_value!(builder, i, BooleanBuilder, as_bool, value),
       ...
  }

If there was a trait that contained append_value and append_null this macro could be replaced with a generic function, which would be a much better developer experience.

Thanks for the library!

[tustvold]

As of the most recent arrow release, the builders implement Extend is this possibly what you are looking for?

[dnsco]

So the builders implement extend but an &mut Builder does not have an implementation. I'm bashing my head against the wall trying to figure out how to make it work.

here's the new method:

pub fn set_val<T, R, V, F>(mut builder: T, value: Option<V>, getter: F) -> Result<()>
where
    T: Extend<Option<R>>,
    V: Deref<Target = Value>,
    F: FnOnce(&V::Target) -> Option<R>,
{
    let v = value
        .as_deref()
        .map(|v| {
            getter(v).ok_or_else(|| {
                let msg = format!("Could not cast {} to correct type", v);
                Error::new(InvalidData, msg)
            })
        })
        .transpose()?;
    builder.extend(vec![v]);
    Ok(())
}

called as:

DataType::Float64 => {
            let b = struct_builder.field_builder::<Float64Builder>(i).unwrap();
            set_val(b, value, Value::as_f64)?
}

Yielding the error:

96 |             set_val(b, value, Value::as_f64)?
   |             ------- ^ the trait `Extend<std::option::Option<_>>` is not implemented for `&mut PrimitiveBuilder<Float64Type>`
   |             |
   |             required by a bound introduced by this call
   |
   = help: the trait `Extend<std::option::Option<<P as ArrowPrimitiveType>::Native>>` is implemented for `PrimitiveBuilder<P>`
note: required by a bound in `set_val`
  --> project/src/record_conversion/builder_appending.rs:64:8
   |
62 | pub fn set_val<T, R, V, F>(mut builder: T, value: Option<V>, getter: F) -> Result<()>
   |        ------- required by a bound in this
63 | where
64 |     T: Extend<Option<R>>,
   |        ^^^^^^^^^^^^^^^^^ required by this bound in `set_val`

Am I missing something simple? If I try and reference *b at the callsite I get:

error[E0507]: cannot move out of `*b` which is behind a mutable reference
  --> prost-arrow/src/record_conversion/builder_appending.rs:96:21
   |
96 |             set_val(*b, value, Value::as_f64)?
   |                     ^^ move occurs because `*b` has type `PrimitiveBuilder<Float64Type>`, which does not implement the `Copy` trait

Is there a reason there can't be an implementation of Extend for &mut Builders?

Thanks!

[dnsco]

Per conversation on discord I was missing an &mut T in the function's signature, extend should do what I need!