zstd compression in write_ipc is incompatible with standalone zstd?

[indigoviolet]

Polars version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of polars.

Issue Description

It seems that writing an IPC file with compression=zstd leads to a file that cannot be decompressed by the standalone zstd program. And an IPC file compressed by the standalone zstd program cannot be read by read_ipc

Reproducible Example

import polars as pl
df = pl.DataFrame({"a": [1,2]})
df.write_ipc("test1.arrow.zst", compression="zstd")
df.write_ipc("test2.arrow")

In [24]:
! source ../../.env && zstd --version && zstd -d test1.arrow.zst
! source ../../.env && zstd test2.arrow
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
zstd: test1.arrow.zst: unsupported format <---------------------------------------- ERROR!!!
test2.arrow          : 39.20%   (   528   B =>    207   B, test2.arrow.zst)


In [23]:
pl.read_ipc("test1.arrow.zstd").frame_equal(pl.read_ipc("test2.arrow"))
could not mmap compressed IPC file, defaulting to normal read
Out [23]:
True

In [25]:
pl.read_ipc("test2.arrow.zst")

Truncated Traceback (Use C-c C-$ to view full TB):
File ~/dev/instant-science/t5/.venv/lib/python3.8/site-packages/polars/internals/dataframe/frame.py:788, in DataFrame._read_ipc(cls, file, columns, n_rows, row_count_name, row_count_offset, rechunk, memory_map)
    786 projection, columns = handle_projection_columns(columns)
    787 self = cls.__new__(cls)
--> 788 self._df = PyDataFrame.read_ipc(
    789     file,
    790     columns,
    791     projection,
    792     n_rows,
    793     _prepare_row_count_args(row_count_name, row_count_offset),
    794     memory_map=memory_map,
    795 )
    796 return self

ArrowErrorException: OutOfSpec("InvalidHeader")

Expected Behavior

I would have expected that these were interchangeable

Installed Versions

``` ---Version info--- Polars: 0.14.9 Index type: UInt32 Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.10 Python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) [GCC 10.3.0] ---Optional dependencies--- pyarrow: 7.0.1 pandas: 1.5.0 numpy: 1.23.3 fsspec: 2022.8.2 connectorx: xlsx2csv: pytz: 2022.2.1

</details>

[indigoviolet]

also woo! #5000

[ritchie46]

also woo! #5000

LOL. Congrats! ;)

[ritchie46]

I don't believe the compression is done on the final byte blob, but more on sub-blobs.

By that I mean that the IPC-file probably still has metadata/ipc file structure that is not compressed, but does have certain data chunks that are compressed.

@jorgecarleitao is this correct?

[indigoviolet]

also woo! #5000

LOL. Congrats! ;)

Well, congrats to you :)

[ghuls]

I don't believe the compression is done on the final byte blob, but more on sub-blobs.

By that I mean that the IPC-file probably still has metadata/ipc file structure that is not compressed, but does have certain data chunks that are compressed.

@jorgecarleitao is this correct?

Yeah, that is correct as far as I know. Only columns are compressed, not the file as whole.

[indigoviolet]

It seems worth documenting this difference. I guess there is a performance benefit that a subset of columns can be decompressed while reading?

[jorgecarleitao]

What @ghuls wrote. The rationale is that it allows

• projection pushdown without having to decompress the whole chunk
• consume the file in chunks without having to decompress the whole thing.
• read the file metadata without having to decompress the whole file

Congrats everyone for the 5k!

[arturdaraujo]

I'm having the same problem

df = pl.read_ipc(r"..\NASDAQ\AACG.feather")
could not mmap compressed IPC file, defaulting to normal read

[kylebarron]

@arturdaraujo That should be just a warning, not an error, and I think is unrelated to this issue

[stinodego]

Closing this as it's not a bug. A feature request to change the functionality is welcome (I believe it already exists in the form of #9283).