Filter out low-scoring skeletons spawned during inference

[olinesn]

Problem: We only allow for filtering based on instance score during tracking. We should have options for filtering even without tracking (e.g., during the HITL labeling process).

Solution: Add Predictor options and associated CLI flags and GUI fields to allow for filtering by score during inference independently of tracking.

This should also allow for filtering based on number of animals in addition to an absolute score threshold. This would cover the use case of doing single-animal tracking using multi-animal models.

See #640 for the first point related to including background frames.

---

Original issue:

As far as I understand, only labelled frames with skeletons are included in the training set. I have a prep where I occasionally have to reach my hand into the cage while the camera is on, and SLEAP inference ends up spawning lots of tracks that are low-scoring skeletons on my glove/arm.

What's the best way to deal with this? Options I can think of include:

1: Is there a way to include frames in the training set that do NOT have any animals in them, to teach the model what's NOT an animal?

2: Can I set a threshold for skeleton score when running inference on a video, and remove all skeletons below a threshold BEFORE they get stitched into tracks?

Thanks!

[olinesn]

This could potentially merged with #640

[talmo]

Hi @olinesn,

1. I don't remember if we tested this, but one workaround could be to have a skeleton with all nodes marked as not visible. I think this might be sufficient to have the frame included as a "negative" example. (This will not work, see comment below)

2. Not quite the same, but using the sleap-track options --tracking.target_instance_count 3 --tracking.pre_cull_to_target 1 without setting the IOU will filter instances by score before tracking them. Not ideal though as it can still include spurious instances as long as it's less than the target count.

We'll work on these as enhancements. Thanks!!

Talmo

[roomrys]

Since we seem to have gotten a lot of cases where people are marking all nodes as not visible, wanted to state here (at the root) that SLEAP will filter out all empty instances (with all non visible points). Labeling an instance as entirely occluded will not do anything (other than use-up your time):

sleap/sleap/nn/data/providers.py

Lines 37 to 59 in 833c2d5

	@classmethod
	def from_user_instances(
	cls, labels: sleap.Labels, with_track_only: bool = False
	) -> "LabelsReader":
	"""Create a `LabelsReader` using the user instances in a `Labels` set.
	Args:
	labels: A `sleap.Labels` instance containing user instances.
	with_track_only: If `True`, load only instances that have a track assigned.
	Useful when training supervised ID models.
	Returns:
	A `LabelsReader` instance that can create a dataset for pipelining.
	Notes:
	This will remove "empty" instances, i.e., instances with no visible points,
	in the original labels. Make a copy of the original labels if needed as they
	will be modified in place.
	"""
	labels.remove_empty_instances(keep_empty_frames=False)
	obj = cls.from_user_labeled_frames(labels)
	obj.user_instances_only = True
	obj.with_track_only = with_track_only
	return obj

[olinesn]

Since we seem to have gotten a lot of cases where people are marking all nodes as not visible, wanted to state here (at the root) that SLEAP will filter out all empty instances (with all non visible points). Labeling an instance as entirely occluded will not do anything (other than use-up your time):

sleap/sleap/nn/data/providers.py

Lines 37 to 59 in 833c2d5

	@classmethod
	def from_user_instances(
	cls, labels: sleap.Labels, with_track_only: bool = False
	) -> "LabelsReader":
	"""Create a `LabelsReader` using the user instances in a `Labels` set.
	Args:
	labels: A `sleap.Labels` instance containing user instances.
	with_track_only: If `True`, load only instances that have a track assigned.
	Useful when training supervised ID models.
	Returns:
	A `LabelsReader` instance that can create a dataset for pipelining.
	Notes:
	This will remove "empty" instances, i.e., instances with no visible points,
	in the original labels. Make a copy of the original labels if needed as they
	will be modified in place.
	"""
	labels.remove_empty_instances(keep_empty_frames=False)
	obj = cls.from_user_labeled_frames(labels)
	obj.user_instances_only = True
	obj.with_track_only = with_track_only
	return obj

Thanks for the heads up!

[roomrys]

#1070 implemented this for top-down models, but this still needs to be applied for other models.