access to "prediction without track identity"

[adpala]

Hi,

Is it possible to get instances labeled as "prediction without track identity" from an ".slp" file?

At the moment I just convert from the .slp to the analysis hdf5 file like this:
sleap-convert --output OUTPUT_FILE --video INPUT_VIDEO --format analysis OUTPUT_FILE.slp

But I haven't found where these instances are stored. I'm interested in these instances, since they correctly fit the animal in those video frames, although for some reason (not the topic of the question) these are not assigned a track.

[roomrys]

Hi @adpala,

Only instances with track identities are saved in the analysis hdf5. As you stated, you will need to navigate within the slp file directly. We have a data structures notebook which gives a few examples for how to navigate the data in your slp.

Basically each slp file is contained within a Labels object.

sleap/sleap/io/dataset.py

Lines 390 to 426 in 57ee08a

	class Labels(MutableSequence):
	"""
	The :class:`Labels` class collects the data for a SLEAP project.
	This class is front-end for all interactions with loading, writing,
	and modifying these labels. The actual storage backend for the data
	is mostly abstracted away from the main interface.
	Attributes:
	labeled_frames: A list of :class:`LabeledFrame` objects
	videos: A list of :class:`Video` objects that these labels may or may
	not reference. The video for every `LabeledFrame` will be
	stored in `videos` attribute, but some videos in
	this list may not have any associated labeled frames.
	skeletons: A list of :class:`Skeleton` objects (again, that may or may
	not be referenced by an :class:`Instance` in labeled frame).
	tracks: A list of :class:`Track` that instances can belong to.
	suggestions: List that stores "suggested" frames for
	videos in project. These can be suggested frames for user
	to label or suggested frames for user to review.
	negative_anchors: Dictionary that stores center-points around
	which to crop as negative samples when training.
	Dictionary key is :class:`Video`, value is list of
	(frame index, x, y) tuples.
	provenance: Dictionary that denotes the origin of the :py:class:`Labels`.
	"""
	labeled_frames: List[LabeledFrame] = attr.ib(default=attr.Factory(list))
	videos: List[Video] = attr.ib(default=attr.Factory(list))
	skeletons: List[Skeleton] = attr.ib(default=attr.Factory(list))
	nodes: List[Node] = attr.ib(default=attr.Factory(list))
	tracks: List[Track] = attr.ib(default=attr.Factory(list))
	suggestions: List[SuggestionFrame] = attr.ib(default=attr.Factory(list))
	negative_anchors: Dict[Video, list] = attr.ib(default=attr.Factory(dict))
	provenance: Dict[Text, Union[str, int, float, bool]] = attr.ib(
	default=attr.Factory(dict)
	)

The Labels object contains a list of LabeledFrames (accessed through Labels.labeled_frames).

sleap/sleap/instance.py

Lines 1282 to 1292 in 57ee08a

	class LabeledFrame:
	"""Holds labeled data for a single frame of a video.
	Args:
	video: The :class:`Video` associated with this frame.
	frame_idx: The index of frame in video.
	instances: List of instances associated with the frame.
	"""
	video: Video = attr.ib()
	frame_idx: int = attr.ib(converter=int)

Each LabeledFrame contains a list of PredictedInstances (accessed through LabeledFrame.predicted_instances).

sleap/sleap/instance.py

Lines 1431 to 1434 in 57ee08a

	@property
	def predicted_instances(self) -> List[PredictedInstance]:
	"""Return list of predicted instances associated with frame."""
	return [inst for inst in self._instances if type(inst) == PredictedInstance]

Each PredictedInstance will have a Track (accessed through PredictedInstance.track). If the Track is None, then the PredictedInstance does not have a track.

sleap/sleap/instance.py

Lines 344 to 367 in 57ee08a

	class Instance:
	"""This class represents a labeled instance.
	Args:
	skeleton: The skeleton that this instance is associated with.
	points: A dictionary where keys are skeleton node names and
	values are Point objects. Alternatively, a point array whose
	length and order matches skeleton.nodes.
	track: An optional multi-frame object track associated with
	this instance. This allows individual animals/objects to be
	tracked across frames.
	from_predicted: The predicted instance (if any) that this was
	copied from.
	frame: A back reference to the :class:`LabeledFrame` that this
	:class:`Instance` belongs to. This field is set when
	instances are added to :class:`LabeledFrame` objects.
	"""
	skeleton: Skeleton = attr.ib()
	track: Track = attr.ib(default=None)
	from_predicted: Optional["PredictedInstance"] = attr.ib(default=None)
	_points: PointArray = attr.ib(default=None)
	_nodes: List = attr.ib(default=None)
	frame: Union["LabeledFrame", None] = attr.ib(default=None)

An (untested) MVP for your use case:

import sleap

ds = "path/to/slp"
labels = sleap.load_file(ds)

lfs = labels.labeled_frames
instances_without_tracks = [inst for inst in lfs.predicted_instances if inst.track is None]

Let us know if you need any extra assistance!

Thanks,
Liezl

[adpala]

Thanks a lot Liezl, that was a great answer!