GPU selection and benchmarks

[quantumdot]

Looking for information on what GPUs are recommended for use with SLEAP, particularly in a real-time or closed-loop setting. Are there benchmarks available for SLEAP on different GPUs, and possibly different model configurations?

[talmo]

Hi @quantumdot,

GPUs:
We've done most of our testing on an Titan RTX, but that's way overkill, especially for realtime. Anything with >8 GB of memory should be enough, especially for inference. Any of the 30xx series should have enough CUDA cores to get your realtime latency pretty low.

Benchmarking:

• Our paper has a good bit of results on performance.
• The Methods has a section on speed benchmarking which is helpful as well.

Closed loop:

• DeepLabStream has native support for using SLEAP models for realtime/closed-loop experiments.
• Here's a module for realtime inference using SLEAP that we used for our closed-loop experiments in the Nature Methods (2022) paper.
• Here's a notebook on realtime inference using SLEAP that's in our docs page (scroll to the last few sections).
• We recently added the sleap-export command which generates TensorFlow SavedModels and frozen graphs, which are compatible for use with C++ TensorFlow inference engines and should be convertible to TensorRT for massive performance gains (especially with FLOAT16 quantization).
• We'll soon be releasing a guide on using the new Bonsai-RX SLEAP package which gives you the ability to use SLEAP within a more sophisticated experiment control framework.

In terms of model configurations: that's tricky since it depends heavily on your dataset and accuracy requirements. The rule of thumb is that you want to first get to a model configuration that is accurate enough for your needs, and then from there start cutting down on the number of computations. A few good hyperparameters to tune for performance:

• Approach: Top-down will almost always be faster than any other approach since it crops a bounding box around the animal. This may even be true in single-animal settings if the animal is small relative to the size of the frame.
• Filters and filter rate: Decreasing these will decrease the overall number of parameters in the model, resulting in fewer computations at the expense of reduced capacity.
• Output stride: Increasing this to 4 or 8 can result in significant speed-ups at the cost of resolution/accuracy.

Let us know if you try any of these and have any questions.

Cheers,

Talmo

[quantumdot]

Hi Talmo,

Wow, thanks for the great, detailed answer!

Do you have any experience with professional cards, such as the RTX A2000 through the RTX A5000, or Tesla architectures?

It would be a great thing to have a benchmark script which users could run with a common set of models on a common set of videos and produce sharable benchmark statistics.

[jmdelahanty]

Our lab has several machines with (I'm pretty sure...) A4000 GPUs on board. I'd be happy to try running SLEAP on one to see how it does. I also did some stuff with messing around in benchmarking (maybe not so professionally/best practices) for our lab here with related scripts here.