Exporting and Merging Track IDs for Two-Animal Tracking with Top-Down ID Models

[neugun]

I am tracking two animals (one always on the left and the other on the right). When I use the top-down model (without the ID option, as the top-down ID model performs poorly in predictions), I don't get any tracks in the output. Is there a way to export the results for both animals and manually assign track IDs to the points?

Additionally, when I use the top-down ID model, I sometimes get multiple IDs like 1,1,1,1 or 2,2,2,2 for the same animal. Is there a way to export the results and merge these track IDs for each animal?

[roomrys]

Hi @neugun,

---

When I use the top-down model (without the ID option, as the top-down ID model performs poorly in predictions), I don't get any tracks in the output.

When you run inference, what are you setting for the tracking parameters? I would recommend:

Tracker: simple
Max number of tracks: 2 (number of animals)
Connect Single Track Breaks: true

with the rest of the parameters set to default values.

Originally posted by @roomrys in #1924 (comment)

---

Is there a way to export the results for both animals and manually assign track IDs to the points?

I could write you a script to manually set the Tracks afterwards if you are not having any luck with the temporal tracker (with parameters as specified above).

Originally posted by @roomrys in #1924 (comment)

---

Additionally, when I use the top-down ID model, I sometimes get multiple IDs like 1,1,1,1 or 2,2,2,2 for the same animal.

Yes, the track names are re-used, but each of them is a different Track object. This is explained in #1652 by the inner workings of the ID model (using VideoReader instead of LabelsReader and thus not having access to the Labels.tracks information). If you are predicting across multiple videos, then this is expected/known. Even if you are running multiple inferences, this is expected/known. After running inference, you can manually rename the Tracks to more meaningful names in the Instances dock/table. You can also delete unused tracks with Tracks > Delete Multiple > Unused.

---

Is there a way to export the results and merge these track IDs for each animal?

Since the Tracks are indeed different Track objects (just the names are re-used across videos/inferences), it would be wrong to merge the Tracks for each animal. The scope of a Track is within a Video. Moreover, each inference will create new Track objects and leave the old Track objects unused (if predicting on the entire video - not the case if predicting on just a few frames); hence my suggestion to Tracks > Delete Multiple > Unused.

---

Thanks,
Liezl

[neugun]

Hi Liezl,

Thank you for the detailed explanation.

Regarding your suggestions:

1. For the tracking parameters, I-have set the Tracker to simple, the Max number of tracks to 2, and enable Connect Single Track Breaks. I left the other parameters as default.
2. For exporting results and manually assigning track IDs, a script to manually set the Tracks afterward would be really helpful. I would appreciate it if you could provide that.

As for the issue with multiple IDs (like 1,1,1,1 or 2,2,2,2) for the same animal when using the top-down ID model, I understand that the track names are re-used and that these are different Track objects. I'll take your advice on renaming the Tracks in the Instances dock/table and managing unused tracks accordingly. I'll refrain from merging Track IDs across videos, as you've pointed out the importance of keeping them as distinct Track objects per video.

Thanks again for your help!

Best,
Neugun

[neugun]

Hi Liezl,

I hope you're doing well. I wanted to update you on the tracking parameters I've been working with for the top-down model. Yesterday, I tried setting the Tracker to "simple," the Max Number of Tracks to 2, and enabled "Connect Single Track Breaks," while leaving the other parameters at their default settings. The results are different this time—I can now find the tracks in the H5 files, but across 8 videos, I ended up with hundreds of track names.

You previously mentioned that after running inference, tracks can be manually renamed to more meaningful names in the Instances dock/table. However, with over 50 tracks in a single video, manual renaming doesn't seem feasible. Is there a way to rename the tracks using code rather than doing it manually? My current strategy is to remove all tracks and use other methods to handle the data.

Additionally, for exporting results and manually assigning track IDs, would you be able to help by writing a script to manually set the track IDs? I've noticed that only half of the points (for one animal, not both) were extracted in the H5 file, so it seems this script would need to work with the SLP files rather than H5.

I've been working on this for several months, encountering different problems each time. About six months ago, I had to set it aside, but I'm determined to solve it now. This time, I'm focusing on getting the correct results using the top-down model and ensuring accurate keypoint tracking.

Thanks so much for your help.

Best regards,
Neugun

[roomrys]

Hi @neugun,

---

Yesterday, I tried setting the Tracker to "simple," the Max Number of Tracks to 2, and enabled "Connect Single Track Breaks," while leaving the other parameters at their default settings. The results are different this time—I can now find the tracks in the H5 files, but across 8 videos, I ended up with hundreds of track names.

If Max Number of Tracks is set to 2, then we should end up with a maximum of 2 tracks per video, so 16 tracks maximum. Are you still using the ID model for inference? If so, do not use the ID model as it will also add tracks which will just be unused when running temporal tracking post-inference.

---

You previously mentioned that after running inference, tracks can be manually renamed to more meaningful names in the Instances dock/table. However, with over 50 tracks in a single video, manual renaming doesn't seem feasible.

This is way too many tracks (especially when we expect only 2 Tracks per video). Please:

1. Save As.... your project to a new name

2. Tracks > Delete Multiple... > All

3. Save

4. Re-run inference setting

   Pipeline: top-down (**not** ID)
   Max instances: 2
   

   and the aforementioned tracking parameters:

   Tracker: simple
   Max number of tracks: 2 (number of animals)
   Connect Single Track Breaks: true
   

   .

---

Additionally, for exporting results and manually assigning track IDs, would you be able to help by writing a script to manually set the track IDs? I've noticed that only half of the points (for one animal, not both) were extracted in the H5 file, so it seems this script would need to work with the SLP files rather than H5.

The H5 files organizes data based on the Track that it is assigned to. Thus, if there is missing data, it points to the animal not being assigned a Track in that particular LabeledFrame.

I am writing the script for you right now (although I believe that we can get the temporal tracker working for you).

---

Thanks,
Liezl

[roomrys]

Here is a script (also shown below) that reassigns tracks based on an x-midpoint. You will need to adjust the dataset path ds to point to your slp file (see TODO) and may also need to adjust the midpoint (a number between 0 and 1 that determines the threshold for left/right).

import sleap
from sleap import Track
# from sleap.gui.app import main as sleap_label

def main(ds: str, ds_output: str, midpoint:float = 0.5):
    """Reassigns tracks based on x-threshold and saves to `ds_output`.

    Note: This function assumes that the dataset has only two animals. No restrictions 
        are in place to ensure that two animals in the same frame are assigned different
        tracks. This is a simple example to demonstrate how to reassign tracks based on
        a threshold.

    Args:
        ds: Path to dataset.
        ds_output: Path to save output dataset.
        midpoint: Threshold for x-coordinate.
    """

    # Load labels
    labels = sleap.load_file(ds)

    # Remove all existing tracks
    labels.remove_all_tracks()

    # Create a new track
    def create_track(name, video):
        track = Track(name=name)
        labels.add_track(video=video, track=track)
        return track

    def name_track(video, side):
        base_name = "mouse"
        video_idx = labels.videos.index(video)
        track_name = f"{base_name}-{side}-{video_idx}"
        return track_name

    def create_left_right_tracks(video):

        # Create left track
        track_left_name = name_track(video=video, side="left")
        track_left = create_track(name=track_left_name, video=video)

        # Create right track
        track_right_name = name_track(video=video, side="right")
        track_right = create_track(name=track_right_name, video=video)
        return track_left, track_right

    # Create a new track for each instance depending on side of x-threshold
    tracks_by_video = {}
    for labeled_frame in labels.labeled_frames:

        # Get video
        video = labeled_frame.video
        if video not in tracks_by_video:
            video_idx = labels.videos.index(video)
            tracks_by_video[video] = create_left_right_tracks(video=video)

        # Get tracks for video
        track_left, track_right = tracks_by_video[video]

        # Threshold for x-coordinate
        width = video.width
        x_threshold = width * midpoint

        for instance in labeled_frame.instances:
            # Get center of instance
            centroid_x = instance.centroid[0]
            if centroid_x < x_threshold:
                instance.track = track_left
            else:
                instance.track = track_right

    # Save the modified labels
    labels.save("output.slp")


if __name__ == "__main__":

    # Load dataset
    ds = r"path/to/dataset.slp"  # TODO: Change this to the path of your dataset

    # Assign left and right tracks
    main(ds=ds, ds_output="output.slp", midpoint=0.5)

    # Open labels file in GUI (optional)
    # sleap_label(["output.slp"])

Thanks,
Liezl

[neugun]

Hi Liezl,

Thanks a lot for the code.
Now there is an error comes up when run the code:

ImportError                               Traceback (most recent call last)
Cell In[222], line 3
      1 import sleap
      2 from sleap import Track
----> 3 from sleap.gui.app import main as sleap_label
      5 def main(ds: str, ds_output: str, midpoint:float = 0.5):
      6     """Reassigns tracks based on x-threshold and saves to `ds_output`.
      7 
      8     Note: This function assumes that the dataset has only two animals. No restrictions 
   (...)
     16         midpoint: Threshold for x-coordinate.
     17     """

File ~\Anaconda3\envs\sleap\lib\site-packages\sleap\gui\app.py:61
     59 import sleap
     60 from sleap.gui.color import ColorManager
---> 61 from sleap.gui.commands import CommandContext, UpdateTopic
     62 from sleap.gui.dialogs.filedialog import FileDialog
     63 from sleap.gui.dialogs.formbuilder import FormBuilderModalDialog

File ~\Anaconda3\envs\sleap\lib\site-packages\sleap\gui\commands.py:48
     46 from sleap.gui.dialogs.delete import DeleteDialog
     47 from sleap.gui.dialogs.filedialog import FileDialog
---> 48 from sleap.gui.dialogs.importvideos import ImportVideos
     49 from sleap.gui.dialogs.merge import MergeDialog, ReplaceSkeletonTableDialog
     50 from sleap.gui.dialogs.message import MessageDialog

File ~\Anaconda3\envs\sleap\lib\site-packages\sleap\gui\dialogs\importvideos.py:35
     26 from qtpy.QtWidgets import (
     27     QPushButton,
     28     QButtonGroup,
   (...)
     31     QComboBox,
     32 )
     34 from sleap.gui.widgets.video import GraphicsView
---> 35 from sleap.io.video import (
     36     Video,
     37     MediaVideo,
     38     HDF5Video,
     39     NumpyVideo,
     40     ImgStoreVideo,
     41     SingleImageVideo,
     42     available_video_exts,
     43 )
     44 from sleap.gui.dialogs.filedialog import FileDialog
     46 import h5py

ImportError: cannot import name 'available_video_exts' from 'sleap.io.video' (C:\Users\zhenggang\Anaconda3\envs\sleap\lib\site-packages\sleap\io\video.py)

Best,
Zhenggang

[roomrys]

Hi @neugun,

I have no clue why your environment is having trouble importing a line that exists. Unless you are using an old sleap environment on new source code?

But, anyway, the import line that is causing the error is not necessarily needed for the functionality - it was just to automatically open the re-tracked labels for you. I updated the code block above to comment out the two lines where this is used.

Thanks,
Liezl

[neugun]

I also noticed the new Sleap can export CSV files, maybe I can apply this to CSV file and solve the track issue? Thanks, Neugun

[roomrys]

Hi @neugun,

I really think that the temporal tracker can work for you with the correct settings. Would you mind uploading your SLP file (with pose estimations on all frames/videos) here (I don't need the videos)? If possible, I would like to fiddle with the tracker on your data to get it working for you.

Thanks,
Liezl

[neugun]

Thanks, Liezl! I have just uploaded a SLP file.

[roomrys]

Hi @neugun,

Wait, your results look pretty good. The "errors in tracking" are actually due to pose estimation on the same animal twice - not due to the tracker. You can try increasing the Sigma Value on the anchor node when running inference - I believe two centroids are being predicted with high confidence on the same animal. Increasing the sigma should eliminate any centroid predictions that are too close to one another. Maybe try a sigma value of 5? I am interested in the results - please keep me updated.

If there are errors in any other videos, you can upload the slp in the form above - let me know if/when you upload.

Thanks,
Liezl

---

These are the steps I took and my findings:

I opened your slp file. There was only pose estimations for a single video.

I noticed that all your frames already had Tracks assigned.

I checked how many tracks there were in your project:
Click on instance, then Tracks > Set Instance Track
and saw only two tracks “track_0” and “track_1”.

You mentioned that there is a left/right barrier, so I set
View > Trail Length > 250
and “scrubbed” through the frames (I used Ctrl + J to jump to all possible integer multiples of 250 and then manually used the left arrow to scrub backwards and find the exact frame) to see if there are any left/right switches.

These are the possible switches that either need manual correction (“proofreading”) or an increased sigma value when running inference:

Frame 724

Frame 1,745

Frame 2,691

Frame 4,515

Frame 4,751

Frame 4,752

Frame 5,999

Frame 6,327

[neugun]

Thanks! Looks like I can only retrain the model to try a sigma value of 5? If I select the trained model, it will not allow me to change the sigma value. Best, Neugun

[roomrys]

Yes, you will need to retrain the model to adjust the sigma value.