Optionally return empty LabeledFrames when no predictions within a frame

[iteal]

Hi!

I have an issue with the BottomupPredictor for one of my videos. I get the following warning during inference:

2020-09-04 15:22:04.048511: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at iterator_ops.cc:941 : Invalid argument: {{function_node __inference_Dataset_map_group_instances_2271}} Tried to stack elements of an empty list with non-fully-defined element_shape: <unknown>
	 [[{{node TensorArrayV2Stack_1/TensorListStack}}]]
INFO:sleap.nn.inference:ERROR in sample index 8989
INFO:sleap.nn.inference:{{function_node __inference_Dataset_map_group_instances_2271}} Tried to stack elements of an empty list with non-fully-defined element_shape: <unknown>
	 [[{{node TensorArrayV2Stack_1/TensorListStack}}]]

This is my code:

    # crops is a numpy array such as len(crops) = 11123
    crops_filepath = os.path.join(temp_dir, "crops.h5")
    with h5py.File(crops_filepath, "w") as f:
        f.create_dataset("frames", data=crops)

    video_reader = VideoReader.from_filepath(filename=crops_filepath, dataset="frames", input_format="channels_last",)

    predictor = BottomupPredictor.from_trained_models(model_path)

    predictions = predictor.predict(video_reader)
    # here len(predictions) = 11118, doesn't match the input
    # predictions[0].frame_idx = 0 but:
    # predictions[8989].frame_index = 8990

The problem is the length of the result predictions don't match the length of my input...
I think I can use "frame_idx" in the LabeledFrame object to find out which frame corresponds to what, but this is not super user friendly.
Would it be possible to modify the implementation to return the same amount of LabeledFrame as inputs? Even if the LabeledFrame instances are empty? And maybe explain what this "Tried to stack elements" error mean too?

Thanks!

[talmo]

Hi @iteal,

So regarding the first error: this will be fixed in 1.1 along with a variety of other fixes related to inference. This happens when no instances are found on a particular frame, but in the future we'll just skip them.

The second point: Sure, we can add an option like that. If you make them into a sleap.Labels dataset instead of a list of frames (by passing make_labels=True to predictor.predict() or predictions = sleap.Labels(predictions)), you'll have a bunch of utilities to make it easier to find frames, like sleap.Labels.find().

We'll add some more tutorials for fancier access patterns with 1.1, but I'll keep this issue open to track the feature request of returning blank LabeledFrames.

[talmo]

Closing because this is now the default in inference in v1.1.0 with a CLI arg to revert it (--no-empty-frames).