parquet format for language-agnostic analysis

[roaldarbol]

There are a few roadblocks to analyse SLEAP data in other languages currently, when relying on the h5 format. I know you're not quite happy with csv, which I agree with, it's not very efficient (and the current csv export is brittle in edge-cases, #1578). But only relying on h5 limits the use of SLEAP, because it is so particular to Python (there are bridges like rhdf5, but it's very unintuitive to use with R and doesn't play nice with the majority of toolchains).

Generally, I would recommend adding export to parquet format. parquet, and the rest of Arrow, works across every major language used for analysis, e.g. Python, R, Julia, MATLAB, Rust, JavaScript etc. Such a data frame can then be "grouped" by instance and bodypart to keep the file size low.

[talmo]

Personally in favor of this and should be easy to do by wrangling the exported data into a pandas dataframe first.

Using sleap-io, here's a minimal implementation in a Colab notebook.

[roaldarbol]

Yeah, would be great! I guess the data structure used for the csv files can be used, so it's really just about writing to .parquet instead of .csv with pyarrow. (I've also done it in R myself, so all good, thanks for the suggestion!).