Failure to track on zoomed-out videos

[milicicn212]

Hi all, I've been using SLEAP to track swim bouts in individual larval fish, and it's generally been doing very well on videos containing a single bout, which are pretty zoomed in. However, when I've tried to predict on larger videos - still containing one fish, but zoomed out to show more of the tank so I can capture multiple bouts in sequence - SLEAP is completely unable to track the fish beyond a certain point. Any points that are visible are highly inaccurate, and the hidden points are scattered randomly, not approximating the "average" skeleton at all.

I was wondering if anyone has any information on why this is happening (maybe something to do with the receptive fields?) and what might be done to circumvent it. I was thinking just adding a few videos at a larger zoom into the dataset might fix it, or maybe messing with some hyperparameters. Thanks so much for any help you can provide, and let me know if you need more information!

[talmo]

Hi @milicicn212,

Sorry for the delay, I think I was the one who was responsible for support when you posted your question 😅

Can you give us some more information about your setup and what you've tried so far?

What's the model configuration look like?

Can you share some screenshots of the zoomed out data?

Are you including the zoomed in videos in the same project as the zoomed out ones? (You probably shouldn't -- I can foresee some problems if they're mixed.)

Messing with some hyperparams will probably be the move here, but let's start by figuring out where you're at :)

Cheers,

Talmo

[milicicn212]

No problem! Apologies for my own delay.

I'm running this network on SLEAP v1.3.3. I haven't tested anything super thoroughly yet. I've just noticed that there seems to be some sort of threshold past which the network totally fails to track the fish anymore. The network is trained pretty much exclusively on videos of individual swim bouts, though I do think there are a couple dozen training frames from more zoomed-out videos. The dimensions of the training videos themselves vary a good deal, as the fish swim different distances during different bouts.

I've attached screenshots of the hyperparameters, a more zoomed-in video, and a zoomed-out video. Let me know if I can provide any more details.