NDArray generated in C++ failing to open in Python

[leoyala]

Hello, I am creating an NDArray of shape (N_images, W, H) in one of my libraries (sample code below). My test runs without a problem, and I can see the file being generated. However, when I try opening it from Python, I get the following error. I am not sure if I am doing something wrong during the creating of the data. I would appreciate it if you could help me to spot what I am doing wrong :).

System information:

Ubuntu 22.04
c-blosc2: 2.13.2 (built from source & all tests passing)
blosc2(python): 2.5.1 (works if I generate arrays from Python)
gcc: 11.4.0
cmake: 3.27.0

Python code:

import blosc2
a = blosc2.open("test_image_dataset.b2nd", "r")
a[0, ...]

Python error:

File ...python3.10/site-packages/blosc2/ndarray.py:120, in NDArray.__getitem__(self, key)
    115 # arr = np.zeros(shape, dtype=self.dtype)
    116 # Benchmarking shows that empty is faster than zeros
    117 # (besides we don't need to fill padding with zeros)
    118 arr = np.empty(shape, dtype=self.dtype)
--> 120 return super().get_slice_numpy(arr, key)

File blosc2_ext.pyx:2048, in blosc2.blosc2_ext.NDArray.get_slice_numpy()

File blosc2_ext.pyx:1911, in blosc2.blosc2_ext._check_rc()

RuntimeError: Error while getting the buffer

Sample code:

#include <xiApi.h>
#include <gtest/gtest.h>
#include <blosc2.h>
#include <b2nd.h>


#define HandleBLOSCResult(res, place) \
    if (res != 0) { \
        std::stringstream errormsg; \
        errormsg << "Error after " << place << " " << res << "\n"; \
        throw std::runtime_error(errormsg.str()); \
    }


TEST(BLOSC, BloscAppend) {
    blosc2_init();
    auto* image = new XI_IMG;
    int width = 4 * 512;
    int height = 4 * 272;
    image->width = width;
    image->height = height;
    image->bp = new unsigned char[image->width * image->height * 2]();
    int N_images = 10;

    char *urlpath = strdup("test_image_dataset.b2nd");
    blosc2_remove_urlpath(urlpath);

    blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS;
    cparams.typesize = sizeof(UINT16);
    cparams.compcode = BLOSC_ZSTD;
    cparams.filters[BLOSC2_MAX_FILTERS - 2] = BLOSC_BITSHUFFLE;
    cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_SHUFFLE;
    cparams.clevel = 5;
    cparams.nthreads = 4;

    blosc2_storage storage = BLOSC2_STORAGE_DEFAULTS;
    storage.contiguous = true;
    storage.cparams = &cparams;
    storage.urlpath = urlpath;

    // Shape of the ndarray
    int64_t shape[] = {N_images, image->height, image->width};
    // Set Chunk shape and BlockShape as per your requirement.
    int32_t chunk_shape[] = {1, height, width};
    int32_t block_shape[] = {1, height, width};

    b2nd_context_t *ctx = b2nd_create_ctx(&storage, 3, shape, chunk_shape, block_shape, NULL, 0, NULL, 0);

    blosc2_storage storage_single_image = BLOSC2_STORAGE_DEFAULTS;
    storage_single_image.cparams = &cparams;
    b2nd_context_t *ctx_single_image = b2nd_create_ctx(&storage_single_image, 3, shape, chunk_shape, block_shape, NULL, 0, NULL, 0);

    // loop through all images
    for(int i = 0; i < N_images; i++) {
        printf("Saving image #: %d\n", i);
        // Generate random image data
        for(int j = 0; j < image->width * image->height; j++){
            reinterpret_cast<uint16_t*>(image->bp)[j] = rand() % 65536; // generate random pixels
        }

        // Determine the buffer size (in bytes)
        const int64_t buffer_size = image->width * image->height * sizeof(uint16_t);

        b2nd_array_t *src;
        int result;
        if (i==0) {
            result = b2nd_full(ctx, &src, image->bp);
            HandleBLOSCResult(result, "b2nd_full");
        } else {
            result = b2nd_open(urlpath, &src);
            HandleBLOSCResult(result, "b2nd_append");

            result = b2nd_append(src, image->bp, buffer_size, 0);
            HandleBLOSCResult(result, "b2nd_append");
        }
        result = b2nd_free(src);
        HandleBLOSCResult(result, "b2nd_free");
    }

    b2nd_free_ctx(ctx);
    b2nd_free_ctx(ctx_single_image);
//    blosc2_remove_urlpath(urlpath);
    blosc2_destroy();
    delete[] static_cast<unsigned char*>(image->bp);
    delete image;
}

[FrancescAlted]

Hi Leonardo. Fortunately, there is a solution that is simpler than your attempts. First you need to create the b2nd array, with nothing inside (b2nd_empty()), then set the images one by one (with b2nd_set_slice_cbuffer()).

I did a small example showing these steps in: https://github.com/Blosc/c-blosc2/blob/main/examples/b2nd/example_stack_images.c

The metadata for the output can be read normally with msgpack tools:

francesc@Francescs-MacBook-Air ~/b/c-blosc2 (main)> ll test_image_dataset.b2nd                            (python-blosc2)
-rw-r--r--  1 francesc  staff    43M Mar 29 16:56 test_image_dataset.b2nd
francesc@Francescs-MacBook-Air ~/b/c-blosc2 (main)> msgpack2json -Bi test_image_dataset.b2nd              (python-blosc2)
["b2frame\u0000",184,44565103,"\u0012\u0000P\u0003",44564480,44564800,2,4456448,4456448,4,1,false,"ext:6:base64:AAAAAAABAAAAAAAAAAAAAA==",[17,{"b2nd":107},["lwADk9MAAAAAAAAACtMAAAAAAAAEQNMAAAAAAAAIAJPSAAAAAdIAAARA0gAACACT0gAAAAHSAAAEQNIAAAgAANsAAAADfHUy"]]]⏎
francesc@Francescs-MacBook-Air ~/b/c-blosc2 (main)> dd bs=1 skip=112 count=1000 <  test_image_dataset.b2nd | msgpack2json -B
1000+0 records in
1000+0 records out
1000 bytes transferred in 0.003125 secs (320000 bytes/sec)
[0,3,[10,1088,2048],[1,1088,2048],[1,1088,2048],0,"|u2"]

Or better yet, using the Python wrappers:

francesc@Francescs-MacBook-Air ~/b/c-blosc2 (main)> python -c"import blosc2; a=blosc2.open('test_image_dataset.b2nd'); print(a.info)"
type    : NDArray
shape   : (10, 1088, 2048)
chunks  : (1, 1088, 2048)
blocks  : (1, 1088, 2048)
dtype   : uint16
cratio  : 1.00
cparams : {'blocksize': 4456448,
 'clevel': 5,
 'codec': <Codec.BLOSCLZ: 0>,
 'codec_meta': 0,
 'filters': [<Filter.NOFILTER: 0>,
             <Filter.NOFILTER: 0>,
             <Filter.NOFILTER: 0>,
             <Filter.NOFILTER: 0>,
             <Filter.NOFILTER: 0>,
             <Filter.SHUFFLE: 1>],
 'filters_meta': [0, 0, 0, 0, 0, 0],
 'nthreads': 4,
 'splitmode': <SplitMode.FORWARD_COMPAT_SPLIT: 4>,
 'typesize': 2,
 'use_dict': 0}
dparams : {'nthreads': 4}

As you can see, the size of the file is 43 MB, so actual compression has not been possible, but that's because we have been using random() values in the example.

I hope that, for real images, you can get significant compression ratios by using different combinations of lossless codecs and filters inside Blosc2. If that is not the case, Blosc2 recently has got support for a relatively rich variety of lossy codecs and filters; these should be useful for images, most specially JPEG2000 support.

[leoyala]

Thank you for the example, I have adjusted my code and it seems to work nicely 👍🏼 . I am moving to the next step which is adding metadata corresponding to each image that I am appending. Could you point me to an example on how to do this for NDArrays? The metadata I want to add is simply a string for each image.

[FrancescAlted]

The way to add metadata is via metalayers. You can choose storing them in the header of the frame (when you don't need to change their size), or in the trailer (aka vlmeta, for variable-length). My recommendation is to use vlmeta in the trailer and in case you need more speed of efficiency, move to the header. You can see an example here: https://github.com/Blosc/c-blosc2/blob/main/examples/frame_vlmetalayers.c

You will see that in the vlmeta API you will usually need to pass a schunk as the first param. All the b2nd arrays in Blosc2 have an associated schunk that is accessible from b2nd->sc (in your example, src->sc). The vlmeta example already stores strings, but you can always store other data types by serializing them (msgpack is strongly recommended for that).

[leoyala]

I just relaized that I will also need to implement a solution where I don't know the number of images beforehand, is it possible to modify the shape of the ndarray after creation?
The motivation for this is that in many ocasions, I will have store every incoming image, and stop doing that at some undetermined time.

[FrancescAlted]

Yes. B2ND arrays can be resized (both enlarged and shrinked) easily with b2nd_resize.

[FrancescAlted]

While trying to do the same with b2nd_append() (which should be closer to your needs), I realized that there was bug preventing data to be appended to 0-sized arrays (shapes with a 0-dim component). I fixed this in recent C-Blosc2 2.14.0, and I took the opportunity to improve the example; usage of b2nd_append() is shown too. As a bonus, the example is adding strings as vlmeta to b2nd arrays too.

[leoyala]

Thank you for the great support @FrancescAlted.
I have tested it and it seems to work quite nicely for my data. Is it possible to attach metadata to each frame individually? Or only to the super-chunck (entire file in this case) ?
Would the way to go about it be to modify the metadata with blosc2_meta_update after appending each image to add the new metadata to the entire file?

[FrancescAlted]

A frame in Blosc2 is just the serial representation of the super-chunk abstraction. What you can do is to add an unlimited number of vlmeta with blosc2_vlmeta_add(). Or use blosc2_meta_update() if what you want is to replace the old content; as vlmeta is at the end of the frame, this is very efficient.

[leoyala]

In case someone else searches for this in the future. This is how I am appending the metadata at the moment.

Dependency:

sudo apt install libmsgpack-dev

Exemplary use:

PackAndAppendMetadata(src, "exposure_us", static_cast<int>(17));
PackAndAppendMetadata(src, "acq_nframe", static_cast<std::string>("sampleString"));

PackAndAppendMetadata:

#include <blosc2.h>
#include <b2nd.h>
#include <msgpack.hpp>

template<typename T>
void PackAndAppendMetadata(b2nd_array_t *src, const char *key, const T& data) {
    // create an array and pack it.
    std::vector<T> dataArray = {data};
    msgpack::sbuffer sbuf;
    msgpack::pack(sbuf, dataArray);
    AppendBLOSCVLMetadata(src, key, sbuf);
}

void AppendBLOSCVLMetadata(b2nd_array_t *src, const char *key, msgpack::sbuffer &newData){
    // Get the existing data
    uint8_t *content = nullptr;
    int32_t content_len;
    int result;
    int metadataExists = blosc2_vlmeta_exists(src->sc, key);
    if (metadataExists < 0){
        result = blosc2_vlmeta_add(src->sc, key,
                                   reinterpret_cast<uint8_t *>(newData.data()), newData.size(),
                                   nullptr);
        if (result < 0){
            throw std::runtime_error("Error when using blosc2_vlmeta_add");
        }
        return;
    } else {
        result = blosc2_vlmeta_get(src->sc, key, &content, &content_len);
        if (result < 0){
            throw std::runtime_error("Error when using blosc2_vlmeta_get");
        }

        // Unpack the existing data
        msgpack::unpacker oldpac;
        oldpac.reserve_buffer(content_len);
        memcpy(oldpac.buffer(), content, content_len);
        oldpac.buffer_consumed(content_len);

        // Unpack the new data
        msgpack::unpacker newpac;
        newpac.reserve_buffer(newData.size());
        memcpy(newpac.buffer(), newData.data(), newData.size());
        newpac.buffer_consumed(newData.size());

        msgpack::object_handle oldoh;
        oldpac.next(oldoh);

        msgpack::object_handle newoh;
        newpac.next(newoh);

        msgpack::sbuffer sbuf;

        // Assume that the existing and new data should have the same type
        if (oldoh.get().type == msgpack::type::ARRAY && oldoh.get().via.array.size > 0 && oldoh.get().via.array.ptr[0].type == msgpack::type::STR) {
            // It's a vector of strings
            auto oldData = oldoh.get().as<std::vector<std::string>>();
            auto appendData = newoh.get().as<std::vector<std::string>>();

            oldData.insert(oldData.end(), appendData.begin(), appendData.end());

            msgpack::pack(sbuf, oldData);
        }
        else {
            // It's a vector of ints
            auto oldData = oldoh.get().as<std::vector<int>>();
            auto appendData = newoh.get().as<std::vector<int>>();

            oldData.insert(oldData.end(), appendData.begin(), appendData.end());

            msgpack::pack(sbuf, oldData);
        }

        // Update the metadata with the new data
        result = blosc2_vlmeta_update(src->sc, key, reinterpret_cast<uint8_t *>(sbuf.data()), sbuf.size(), NULL);
        if (result < 0){
            throw std::runtime_error("Error when using blosc2_vlmeta_update");
        }
    }
}

[leoyala]

A quick update on this for people that might be interested on it. The proposal I posted earlier works well when storing several thousands of images to the same file. However, when the length is too long (e.g. 40,000 images). The method of appending metadata for each single file becomes too slow and there is a considerable delay. This is because on my approach, I am re-writing the entire lists of metadata when each new image arrives. For applications where speed is crucial, I would rather recommend storing the metadata in memory and only save it once at the end.

[FrancescAlted]

Is really your case that writing metadata is slowing down the writing of actual data? This is somewhat unexpected to me, as the general case is that data is much larger in general than metadata (but of course, your mileage may vary and all of that ;-).

[leoyala]

I tested this, and it is indeed the case that the speed after several thousand iterations is reduced considerably. This is, I presume, because in my example I am unpacking and repacking the data on each iteration, which is not ideal. Below is an example to see this effect. I am only appending metadata in this example, and keeping the array data fixed.

#include <blosc2.h>
#include <b2nd.h>
#include <msgpack.hpp>
#include <iostream>
#include <chrono>
#include <stdexcept>


void AppendBLOSCVLMetadata(b2nd_array_t *src, const char *key, msgpack::sbuffer &newData){
    // Get the existing data
    uint8_t *content = nullptr;
    int32_t content_len;
    int result;
    int metadataExists = blosc2_vlmeta_exists(src->sc, key);
    if (metadataExists < 0){
        result = blosc2_vlmeta_add(src->sc, key,
                                   reinterpret_cast<uint8_t *>(newData.data()), newData.size(),
                                   nullptr);
        if (result < 0){
            throw std::runtime_error("Error when using blosc2_vlmeta_add");
        }
        return;
    } else {
        result = blosc2_vlmeta_get(src->sc, key, &content, &content_len);
        if (result < 0){
            throw std::runtime_error("Error when using blosc2_vlmeta_get");
        }

        // Unpack the existing data
        msgpack::unpacker oldpac;
        oldpac.reserve_buffer(content_len);
        memcpy(oldpac.buffer(), content, content_len);
        oldpac.buffer_consumed(content_len);

        // Unpack the new data
        msgpack::unpacker newpac;
        newpac.reserve_buffer(newData.size());
        memcpy(newpac.buffer(), newData.data(), newData.size());
        newpac.buffer_consumed(newData.size());

        msgpack::object_handle oldoh;
        oldpac.next(oldoh);

        msgpack::object_handle newoh;
        newpac.next(newoh);

        msgpack::sbuffer sbuf;

        // Assume that the existing and new data should have the same type
        if (oldoh.get().type == msgpack::type::ARRAY && oldoh.get().via.array.size > 0 && oldoh.get().via.array.ptr[0].type == msgpack::type::STR) {
            // It's a vector of strings
            auto oldData = oldoh.get().as<std::vector<std::string>>();
            auto appendData = newoh.get().as<std::vector<std::string>>();

            oldData.insert(oldData.end(), appendData.begin(), appendData.end());

            msgpack::pack(sbuf, oldData);
        }
        else {
            // It's a vector of ints
            auto oldData = oldoh.get().as<std::vector<int>>();
            auto appendData = newoh.get().as<std::vector<int>>();

            oldData.insert(oldData.end(), appendData.begin(), appendData.end());

            msgpack::pack(sbuf, oldData);
        }

        // Update the metadata with the new data
        result = blosc2_vlmeta_update(src->sc, key, reinterpret_cast<uint8_t *>(sbuf.data()), sbuf.size(), NULL);
        if (result < 0){
            throw std::runtime_error("Error when using blosc2_vlmeta_update");
        }
    }
}


template<typename T>
void PackAndAppendMetadata(b2nd_array_t *src, const char *key, const T& data) {
    // create an array and pack it.
    std::vector<T> dataArray = {data};
    msgpack::sbuffer sbuf;
    msgpack::pack(sbuf, dataArray);
    AppendBLOSCVLMetadata(src, key, sbuf);
}


b2nd_array_t* CreateBLOSCArray(){
    blosc2_init();
    const int32_t width = 4 * 512;
    const int32_t height = 4 * 272;
    const int64_t buffershape[] = {1, height, width};
    // Determine the buffer size of the image (in bytes)
    const int64_t buffersize = width * height * (int64_t)sizeof(uint16_t);
    uint16_t* image = static_cast<uint16_t*>(std::malloc(buffersize * sizeof(uint16_t)));
    int64_t N_images = 10;

    blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS;
    cparams.typesize = sizeof(uint16_t);
    cparams.compcode = BLOSC_BLOSCLZ;
    cparams.clevel = 5;
    cparams.nthreads = 4;

    blosc2_storage storage = BLOSC2_STORAGE_DEFAULTS;
    storage.contiguous = true;  // for a single file in output
    char *urlpath = "example_stack_images_set_slice.b2nd";
    blosc2_remove_urlpath(urlpath);
    storage.urlpath = urlpath;
    storage.cparams = &cparams;

    // shape, chunkshape and blockshape of the ndarray
    int64_t shape[] = {N_images, height, width};
    int32_t chunkshape[] = {1, height, width};
    int32_t blockshape[] = {1, height, width};

    b2nd_context_t *ctx = b2nd_create_ctx(&storage, 3,
                                            shape, chunkshape, blockshape,
                                            "|u2", DTYPE_NUMPY_FORMAT,
                                            NULL, 0);
    b2nd_array_t *src;
    if (b2nd_empty(ctx, &src) < 0) {
        throw std::runtime_error("Could not create empty array");
    }

    printf("Saving images (append version)...\n");
    for (int64_t i = 0; i < N_images; i++) {
        // Generate random image data
        for (int j = 0; j < width * height; j++) {
        image[j] = rand() % 65536; // generate random pixels (uncompressible data)
        }

        if (b2nd_append(src, image, buffersize, 0) < 0) {
            throw std::runtime_error("Could not append array");
        }
    }
    free(image);
    return src;
}

int main() {
    // Start time measurement
    auto start = std::chrono::high_resolution_clock::now();

    const int iterations = 5000;
    const int printInterval = 20000; // Print progress every 1000 iterations

    b2nd_array_t* src = CreateBLOSCArray();

    for (int i = 0; i < iterations; ++i) {
        PackAndAppendMetadata(src, "Key", static_cast<std::string>("sampleString"));

        // Print progress and speed
        if (i % printInterval == 0 && i != 0) {
            auto end = std::chrono::high_resolution_clock::now();
            std::chrono::duration<double> duration = end - start;
            double speed = printInterval / duration.count(); // iterations per second
            std::cout << "Progress: " << i << "/" << iterations
                      << " Speed: " << speed << " iterations/second" << std::endl;
            start = std::chrono::high_resolution_clock::now(); // Reset start time
        }
    }

    // End time measurement and calculate total duration
    auto end = std::chrono::high_resolution_clock::now();
    std::chrono::duration<double> duration = end - start;
    double totalSpeed = iterations / duration.count(); // iterations per second

    std::cout << "Total time: " << duration.count() << " seconds" << std::endl;
    std::cout << "Total speed: " << totalSpeed << " iterations/second" << std::endl;

    b2nd_free(src);
    blosc2_destroy();
    blosc2_remove_urlpath("example_stack_images_set_slice.b2nd");
    return 0;
}

[FrancescAlted]

Ah, so you are enlarging the metadata per every iteration. But I was referring more to the fact that the time for a single metadata insertion should be negligible compared with adding a new image to the b2nd array.

[leoyala]

Ok yes, I agree that the compression of one frame is generally more computationally expensive than the addition of one entry of metadata.

[FrancescAlted]

It is worth mentioning that updating vlmeta does not need the new data to be of the same length than the previous one. This is one of the niceties of vlmeta (as compared with metalayers in the header).