Implement LZ4 compression of blobs

[vmarkovtsev]

Hi! Thanks for this project, it is really cool and we use it to serialize our machine learning models (e.g., https://github.com/src-d/ast2vec). Brilliant idea to combine human-readable YAML and space-efficient blobs.

As I understood from the code and the format specification, ASDF supports zlib and bz2 binary compression. I've recently learned about an exciting compression codec, lz4. As you see from their benchmarks, LZ4 HC is damn slow at compression, yields a similar to zlib compression ratio, but is an order of magnitude faster than zlib during decompression. Our use case is indifferent to the time it takes to generate an ASDF, but is sensitive to the loading lag. I believe, we are not the only ones.

I propose to implement lz4 as the third compression option and include it into some next ASDF specification. I am ready to add it myself, provided by you bless me.

[drdavella]

Hi @vmarkovtsev, it's great to hear that this is a useful format for you, and thanks for the contributions. In principle I don't think there's any issue with supporting additional compression algorithms. However, I wonder if it would make more sense to provide a plug-and-play infrastructure for compression implementations rather than hard-coding additional support into the ASDF library itself. Maybe in the short term we can integrate your changes but in the long term we can provide a more flexible infrastructure.

Also, in the short term, rather than introducing a hard dependency on LZ4, maybe LZ4 support in ASDF should only be available on systems where it is already installed.

[vmarkovtsev]

Ah, I see what you mean, passing a custom class which implements compression/decompression to all_array_compression. I will be more than happy to add this feature in the future!

Regarding lz4, it is already plug and play: import happens lazily, just like with zlib/bzip2. Do you mean smth special?

[drdavella]

Ah I'm sorry, I didn't look at the diffs closely enough. There are a few other tests that are failing as a result of some recent changes to astropy. I would like to fix these before integrating your changes, which may take a little while longer. Do you mind if I hold off on the merge until this is done?

[vmarkovtsev]

Sure, no problem! I will ping here every week 😉

[vmarkovtsev]

As promised, the first ping 😄

[drdavella]

Hey, things are going a little slower than I hoped but I think I should be able to pull this in soon. Maybe today, maybe tomorrow.