Ability to pass empty Arrow tables/datasets to write_deltalake with rust engine

[jorritsandbrink]

Description

It's currently not possible to pass empty Arrow tables or datasets to write_deltalake when using the rust engine.

Write empty Arrow table:

import pyarrow as pa
from deltalake import write_deltalake

arrow_table = pa.Table.from_pydict(
    {"foo": [1, 2], "bar": [True, False]}
)
empty_arrow_table = arrow_table.schema.empty_table()

write_deltalake("my_delta_table", arrow_table, mode="append", engine="rust")  # this creates the Delta table
write_deltalake("my_delta_table", empty_arrow_table, mode="append", engine="rust")  # this errors (but works with `pyarrow` engine)

Error:

DeltaError: Generic error: No data source supplied to write command.

Write empty dataset:

import pyarrow.parquet as pq

pq.write_table(empty_arrow_table, "my_empty_parquet_file.parquet")
empty_arrow_dataset = pa.dataset.dataset("my_empty_parquet_file.parquet")
write_deltalake("my_delta_table", empty_arrow_dataset, mode="append", engine="rust")  # this errors on both `pyarrow` and `rust` engine

Error (when using rust engine, pyarrow throws different error):

PanicException: called `Result::unwrap()` on an `Err` value: CDataInterface("Index error: list index out of range. Detail: Python exception: IndexError")

Use Case
I now use this flow to handle an empty table:

if arrow_table.num_rows > 0:
    write_deltalake(...)

And this flow to handle an empty dataset:

arrow_table = arrow_dataset.to_table()

if arrow_table.num_rows > 0:
    write_deltalake(...)

The dataset case is particulary unpleasant, because you need to eagerly materialize the dataset to a table in memory just to check if it's empty.

It would be nice if we could simply use

write_deltalake(..., data=potentially_empty_arrow_table_or_dataset, ...)

and leave handling of the "empty case" to delta-rs.

Related Issue(s)

• Create empty table failed on rust engine #2354 (only regards initial table creation, not subsequent writes)

[sherlockbeard]

can you share full code example for pyarrow dataset ?.

[jorritsandbrink]

@sherlockbeard yes, here it is:

import pyarrow as pa
import pyarrow.parquet as pq
from deltalake import write_deltalake

arrow_table = pa.Table.from_pydict(
    {"foo": [1, 2], "bar": [True, False]}
)
empty_arrow_table = arrow_table.schema.empty_table()

pq.write_table(empty_arrow_table, "my_empty_parquet_file.parquet")
empty_arrow_dataset = pa.dataset.dataset("my_empty_parquet_file.parquet")

write_deltalake("my_delta_table", arrow_table, mode="append", engine="rust")  # this creates the Delta table
write_deltalake("my_delta_table", empty_arrow_dataset, mode="append", engine="rust")  # this errors on both `pyarrow` and `rust` engine

[conglyne222]

I have been trying to write an empty pandas/polars/pyarrow dataset to delta lake into a Fabric lakehouse. It works well if I run the code written in Python inside a Fabric notebook, but the error Generic error: No data source supplied to write command. occurred in Azure Function (Python V2) with the same code.

[rustyconover]

This still seems to be an issue with deltalake==0.24.0

[conglyne222]

My deltalake version is 0.22.3

[ion-elgreco]

This still seems to be an issue with deltalake==0.24.0

It's very simple to check manually if your incoming data is empty or not. Either way, once we have streamed writing support we won't error out when incoming data is empty

[rustyconover]

Cool! When is the streaming support expected to be released?

…

On Mon, Jan 20, 2025 at 02:23 Ion Koutsouris ***@***.***> wrote: This still seems to be an issue with deltalake==0.24.0 It's very simple to check manually if your incoming data is empty or not. Either way, once we have streamed writing support we won't error out when incoming data is empty — Reply to this email directly, view it on GitHub <#2686 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFSWJL7ZOB3KCBTB6NFKS32LSQADAVCNFSM6AAAAABLELZVXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBRGYYDQNZRG4> . You are receiving this because you commented.Message ID: ***@***.***>