Towards Multi-Modal Animal Pose Estimation: An In-Depth Analysis [Paper]

A repository of unimodal and multi-modal animal pose estimation datasets and papers. This repository is associated with our systematic review, entitled "Towards Multi-Modal Animal Pose Estimation: An In-Depth Analysis".

Authors: Qianyi Deng, Oishi Deb, Amir Patel, Christian Rupprecht, Philip Torr, Niki Trigoni, Andrew Markham.

If you find this work useful for your research, please feel free to cite the paper and leave a ⭐.

@misc{deng2024multimodalanimalposeestimation,
      title={Towards Multi-Modal Animal Pose Estimation: An In-Depth Analysis}, 
      author={Qianyi Deng and Oishi Deb and Amir Patel and Christian Rupprecht and Philip Torr and Niki Trigoni and Andrew Markham},
      year={2024},
      eprint={2410.09312},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2410.09312}, 
}

News

📌 We are continuously monitoring the latest research and encourage contributions to our repository. If your work aligns with our focus, don't hesitate to submit an issue or a pull request. Feedback and contributions are welcome!

Table of Contents

Methods Papers:

• RGB-Based APE
  • 2D RGB-Based APE
    • Single Animal
      • Heatmap-Based
      • Regression-Based
    • Multiple Animals
      • Top-Down
      • Bottom-Up
      • Top-Down&Bottom-Up
  • 3D RGB-Based APE
    • Single Camera View
      • Skeleton-Only Methods
      • Animal Mesh Recovery (Model-Based)
      • Animal Mesh Recovery (Model-Free)
    • Multiple Camera Views
• Non-RGB-Based Unimodal APE
  • IMU-Based APE
• Multi-Modal APE
  • RGB+Depth+Thermal
  • RGB+Depth
  • RGB+LiDAR
  • RGB+Acoustic
  • RGB+Language
  • Animal-Borne Cameras+GPS+IMUs (ego-view)

Datasets Papers:

• RGB-Based APE Dataset
  • Images-Based
  • Videos-Based
• Other-Sensor-Based APE Dataset
  • RGB+MoCap
  • RGB+Depth+MoCap
  • RGB+LiDAR
  • Infrared
  • 3D Scanner
  • MoCap+IMU

Methods Papers

RGB-Based APE

2D RGB-Based APE

Single Animal

Heatmap-Based

• Decompose to generalize: Species-generalized animal pose estimation. Guangrui Li, Yifan Sun, Zongxin Yang, and Yi Yang. ICLR (2023).

• ScarceNet: Animal Pose Estimation with Scarce Annotations. Chen Li and Gim Hee Lee. CVPR (2023).

• A unified framework for domain adaptive pose estimation. Donghyun Kim, Kaihong Wang, Kate Saenko, Margrit Betke, and Stan Sclaroff. ECCV (2022).

• From synthetic to real: Unsupervised domain adaptation for animal pose estimation. Chen Li and Gim Hee Lee. CVPR (2021).

• Deformation-aware unpaired image translation for pose estimation on laboratory animals. Siyuan Li, Semih Gunel, Mirela Ostrek, Pavan Ramdya, Pascal Fua, and Helge Rhodin. CVPR (2020).

• DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. Jacob M Graving, Daniel Chae, Hemal Naik, Liang Li, Benjamin Koger, Blair R Costelloe, and Iain D Couzin. Elife (2019).

• Fast animal pose estimation using deep neural networks. Talmo D Pereira, Diego E Aldarondo, Lindsay Willmore, Mikhail Kislin, Samuel S-H Wang, Mala Murthy, and Joshua W Shaevitz. Nature methods (2019).

• DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Alexander Mathis, Pranav Mamidanna, Kevin M Cury, Taiga Abe, Venkatesh N Murthy, Mackenzie Weygandt Mathis, and Matthias Bethge. Nature neuroscience (2018).

Regression-Based

• Few-shot keypoint detection with uncertainty learning for unseen species. Changsheng Lu and Piotr Koniusz. CVPR (2022).

Multiple Animals

Top-Down

• SuperAnimal pretrained pose estimation models for behavioral analysis. (SuperAnimal-Quadruped) Shaokai Ye, Anastasiia Filippova, Jessy Lauer, and et al. Nature Communications (2024).

• The Hitchhiker’s Guide to Endangered Species Pose Estimation. Jakub Straka, Marek Hruz, and Lukas Picek. WACV (2024).

• DepthFormer: A High-Resolution Depth-Wise Transformer for Animal Pose Estimation. Sicong Liu, Qingcheng Fan, Shanghao Liu, and Chunjiang Zhao. Agriculture (2022).

• Deep-learning-based identification, tracking, pose estimation and behaviour 389 classification of interacting primates and mice in complex environments. Markus Marks et al . Nature Machine (2022).

Bottom-Up

• Multi-animal pose estimation, identification and tracking with DeepLabCut. Jessy Lauer, Mu Zhou, Shaokai Ye, William Menegas, Steffen Schneider, Tanmay Nath, Mohammed Mostafizur Rahman, Valentina Di Santo, Daniel Soberanes, Guoping Feng, et al. Nature Methods (2022).

• Multi-pig Pose Estimation Using DeepLabCut. Fahimeh Farahnakian, Jukka Heikkonen, and Stefan Björkman. IEEE ICICIP (2021).

• Multiple monkey pose estimation using openpose. Salvador Blanco Negrete, Rollyn Labuguen, Jumpei Matsumoto, Yasuhiro Go, Ken-ichi Inoue, and Tomohiro Shibata. bioRxiv (2021).

• Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. CVPR (2017).

Top-Down&Bottom-Up

• SuperAnimal pretrained pose estimation models for behavioral analysis. (SuperAnimal-TopViewMouse) Shaokai Ye, Anastasiia Filippova, Jessy Lauer, and et al. Nature Communications (2024).

• SLEAP: A deep learning system for multi-animal pose tracking. Talmo D Pereira, Nathaniel Tabris, Arie Matsliah, et al . Nature methods (2022).

3D RGB-Based APE

Single Camera View

Skeleton-Only Methods

• DigiDogs: Single-View 3D Pose Estimation of Dogs Using Synthetic Training Data. Moira Shooter, Charles Malleson, and Adrian Hilton. WACV (2024).

• A Horse with no Labels: Self-Supervised Horse Pose Estimation from Unlabelled Images and Synthetic Prior. Jose Sosa and David Hogg. CVPR (2023).

• Unsupervised 3D Animal Canonical Pose Estimation with Geometric Self- Supervision. Xiaowei Dai, Shuiwang Li, Qijun Zhao, and Hongyu Yang. IEEE Automatic Face and Gesture Recognition (2023).

• 3D mouse pose from single-view video and a new dataset. Bo Hu, Bryan Seybold, Shan Yang, Avneesh Sud, Yi Liu, Karla Barron, Paulyn Cha, Marcelo Cosino, Ellie Karlsson, Janessa Kite, et al. Scientific Reports (2023).

• LiftPose3D, a deep learning-based approach for transforming two-dimensional to three-dimensional poses in laboratory animals. Adam Gosztolai, Semih Günel, Victor Lobato-Ríos, Marco Pietro Abrate, Daniel Morales, Helge Rhodin, Pascal Fua, and Pavan Ramdya. Nature methods (2021).

• Three-dimensional pose estimation for laboratory mouse from monocular images. Ghadi Salem, Jonathan Krynitsky, Monson Hayes, Thomas Pohida, and Xavier Burgos-Artizzu. IEEE Transactions on Image Processing (2019).

Animal Mesh Recovery (Model-Based)

• BITE: Beyond priors for improved three-D dog pose estimation. Nadine Rüegg, Shashank Tripathi, Konrad Schindler, Michael J Black, and Silvia Zuffi. CVPR (2023).

• Barc: Learning to regress 3d dog shape from images by exploiting breed information. Nadine Rüegg, Silvia Zuffi, Konrad Schindler, and Michael J Black. CVPR (2022).

• Birds of a feather: Capturing avian shape models from images. Yufu Wang, Nikos Kolotouros, Kostas Daniilidis, and Marc Badger. CVPR (2021).

• Coarse-to-fine animal pose and shape estimation. Chen Li and Gim Hee Lee. NeurIPS (2021).

• Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop. Benjamin Biggs, Oliver Boyne, James Charles, Andrew Fitzgibbon, and Roberto Cipolla. ECCV (2020).

• Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images" In the Wild". Silvia Zuffi, Angjoo Kanazawa, Tanya Berger-Wolf, and Michael J Black. CVPR (2019).

• 3D menagerie: Modeling the 3D shape and pose of animals. Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. CVPR (2017).

Animal Mesh Recovery (Model-Free)

• Learning the 3D Fauna of the Web. Zizhang Li, Dor Litvak, Ruining Li, Yunzhi Zhang, Tomas Jakab, Christian Rupprecht, Shangzhe Wu, Andrea Vedaldi, and Jiajun Wu. CVPR (2024).

• Banmo: Building animatable 3d neural models from many casual videos. Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. CVPR (2022).

• Magicpony: Learning articulated 3d animals in the wild. Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. CVPR (2023).

• Dove: Learning deformable 3d objects by watching videos. Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. IJCV (2023).

Multiple Camera Views

• Multi-animal 3D social pose estimation, identification and behaviour embedding with a few-shot learning framework. Yaning Han, Ke Chen, Yunke Wang, and et al. Nature Machine Intelligence (2024).

• Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL. Liang An, Jilong Ren, Tao Yu, Tang Hai, Yichang Jia, and Yebin Liu. Nature Communications (2023).

• Three-dimensional unsupervised probabilistic pose reconstruction (3D-UPPER) for freely moving animals. Aghileh S Ebrahimi, Patrycja Orlowska-Feuer, Qian Huang, Antonio G Zippo, Franck P Martial, Rasmus S Petersen, and Riccardo Storchi. Scientific Reports (2023).

• Improving 3D Markerless Pose Estimation of Animals in the Wild using Low-Cost Cameras. Naoya Muramatsu, Zico da Silva, Daniel Joska, Fred Nicolls, and Amir Patel. 2022. IEEE IROS (2022).

• Anipose: a toolkit for robust markerless 3D pose estimation. Pierre Karashchuk, Katie L Rupp, Evyn S Dickinson, Sarah Walling-Bell, Elischa Sanders, Eiman Azim, Bingni W Brunton, and John C Tuthill. Cell reports (2021).

• Geometric deep learning enables 3D kinematic profiling across species and environments. Timothy W Dunn, Jesse D Marshall, Kyle S Severson, Diego E Aldarondo, David GC Hildebrand, Selmaan N Chettih, William L Wang, Amanda J Gellis, David E Carlson, Dmitriy Aronov, et al. Nature methods (2021).

• Animal pose estimation from video data with a hierarchical von Mises-Fisher-Gaussian model. Libby Zhang, Tim Dunn, Jesse Marshall, Bence Olveczky, and Scott Linderman. PMLR (2021).

• Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio. Praneet C Bala, Benjamin R Eisenreich, Seng Bum Michael Yoo, Benjamin Y Hayden, Hyun Soo Park, and Jan Zimmermann. Nature Communications (2020).

• Monet: Multiview semi-supervised keypoint detection via epipolar divergence. Yuan Yao, Yasamin Jafarian, and Hyun Soo Park. ICCV (2019).

Non-RGB-Based Unimodal APE

IMU-Based APE

• Wearable inertial sensor-based limb lameness detection and pose estimation for horses. Tarik Yigit, Feng Han, Ellen Rankins, Jingang Yi, Kenneth H McKeever, and Karyn Malinowski. IEEE Transactions on Automation Science and Engineering (2022).

Multi-Modal APE

RGB+Depth+Thermal

• Animal Pose Tracking: 3D Multimodal Dataset and Token-based Pose Optimization. Mahir Patel, Yiwen Gu, Lucas C Carstensen, Michael E Hasselmo, and Margrit Betke. IJCV (2023).

RGB+Depth

• Automated measurement of livestock body based on pose normalisation using statistical shape model. Xinying Luo, Yihu Hu, Zicheng Gao, Hao Guo, and Yang Su. Biosystems Engineering (2023).

• RGBD-Dog: Predicting Canine Pose from RGBD Sensors. Sinead Kearney, Wenbin Li, Martin Parsons, Kwang In Kim, and Darren Cosker. CVPR (2020).

RGB+LiDAR

• WildPose: A Long-Range 3D Wildlife Motion Capture System. Naoya Muramatsu, Sangyun Shin, Qianyi Deng, Andrew Markham, and Amir Patel. BioRxiv (2024).

RGB+Acoustic

• CLHOP: Combined Audio-Video Learning for Horse 3D Pose and Shape Estimation. Ci Li, Elin Hernlund, Hedvig Kjellström, and Silvia Zuffi. arXiv (2024).

• The Sound of Motion: Multimodal horse motion estimation from video and audio. Ci Li, Elin Hernlund, Hedvig Kjellström, and Silvia Zuffi. CVPR Workshop (2022).

RGB+Language

• AWOL: Analysis WithOut synthesis using Language. Silvia Zuffi and Michael J. Black. arXiv (2024).

• AmadeusGPT: a natural language interface for interactive animal behavioral analysis. Shaokai Ye, Jessy Lauer, Mu Zhou, Alexander Mathis, and Mackenzie W Mathis. NeurIPS (2023).

• CLAMP: Prompt-Based Contrastive Learning for Connecting Language and Animal Pose. Xu Zhang, Wen Wang, Zhe Chen, Yufei Xu, Jing Zhang, and Dacheng Tao. CVPR (2023).

Animal-Borne Cameras+GPS+IMUs (ego-view)

• Tracking the cheetah tail using animal-borne cameras, GPS, and an IMU. Amir Patel, Bradley Stocks, Callen Fisher, Fred Nicolls, and Edward Boje. IEEE sensors letters (2017).

Datasets Papers

RGB-Based APE Dataset

Images-Based

• Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shape. Jiacong Xu, Yi Zhang, Jiawei Peng, and et al. ICCV (2023).

• OpenApePose, a database of annotated ape photographs for pose estimation. Nisarg Desai, Praneet Bala, Rebecca Richardson, Jessica Raper, Jan Zimmermann, and Benjamin Hayden. ELife (2023).

• OpenMonkeyChallenge: Dataset and Benchmark Challenges for Pose Estimation of Non-human Primates. Yuan Yao, Praneet Bala, Abhiraj Mohan, and et al. IJCV (2022).

• Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding. Xun Long Ng, Kian Eng Ong, Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu. CVPR (2022).

• MacaquePose: A Novel “In the Wild” Macaque Monkey Pose Dataset for Markerless Motion Capture. Rollyn Labuguen, Jumpei Matsumoto, Salvador Blanco Negrete, Hiroshi Nishimaru, Hisao Nishijo, Masahiko Takada, Yasuhiro Go, Ken-ichi Inoue, and Tomohiro Shibata. Frontiers in Behavioral Neuroscience (2021).

• Ap-10k: A benchmark for animal pose estimation in the wild. Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao. NeurIPS (2021).

• Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop. Benjamin Biggs, Oliver Boyne, James Charles, Andrew Fitzgibbon, and Roberto Cipolla. ECCV (2020).

• Learning From Synthetic Animals. Jiteng Mu, Weichao Qiu, Gregory D. Hager, and Alan L. Yuille. CVPR (2020).

• Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images" In the Wild". Silvia Zuffi, Angjoo Kanazawa, Tanya Berger-Wolf, and Michael J Black. CVPR (2019).

• Cross-domain adaptation for animal pose estimation. Jinkun Cao, Hongyang Tang, Hao-Shu Fang, Xiaoyong Shen, Cewu Lu, and Yu-Wing Tai. ICCV (2019).

• Animals-10 dataset. Corrado Alessio. (2018).

• [Microsoft coco: Common objects in context].(https://cocodataset.org/#download.) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. ECCV (2014).

• Novel Dataset for Fine-Grained Image Categorization. Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. CVPR Workshop (2011).

Videos-Based

• ChimpACT: A Longitudinal Dataset for Understanding Chimpanzee Behaviors. Xiaoxuan Ma, Stephan P. Kaufhold, Jiajun Su, Wentao Zhu, Jack Terwilliger, Andres Meza, Yixin Zhu, Federico Rossano, and Yizhou Wang. NeurIPS (2023).

• Apt-36k: A large-scale benchmark for animal pose estimation and tracking. Yuxiang Yang, Junjie Yang, Yufei Xu, Jing Zhang, Long Lan, and Dacheng Tao. NeurIPS (2022).

• Pretraining boosts out-of-domain robustness for pose estimation. Alexander Mathis, Thomas Biasi, Steffen Schneider, Mert Yuksekgonul, Byron Rogers, Matthias Bethge, and Mackenzie W Mathis. WACV (2021).

• AcinoSet: a 3D pose estimation dataset and baseline models for Cheetahs in the wild. Daniel Joska, Liam Clark, Naoya Muramatsu, Ricardo Jericevich, Fred Nicolls, Alexander Mathis, Mackenzie W Mathis, and Amir Patel. IEEE ICRA (2021).

• Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio. Praneet C Bala, Benjamin R Eisenreich, Seng Bum Michael Yoo, Benjamin Y Hayden, Hyun Soo Park, and Jan Zimmermann. Nature communications (2020).

• ATRW: A Benchmark for Amur Tiger Re-identification in the Wild. Shuyuan Li, Jianguo Li, Hanlin Tang, Rui Qian, and Weiyao Lin. ACM International Conference on Multimedia (2020).

• Creatures great and SMAL: Recovering the shape and motion of animals from video. Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla. ACCV (2018).

• Behavior discovery and alignment of articulated object classes from unstructured video. Luca Del Pero, Susanna Ricco, Rahul Sukthankar, and Vittorio Ferrari. IJCV (2017).

• Articulated motion discovery using pairs of trajectories. Luca Del Pero, Susanna Ricco, Rahul Sukthankar, and Vittorio Ferrari. CVPR (2015).

Other-Sensor-Based APE Dataset

RGB+MoCap

• The Poses for Equine Research Dataset (PFERD). Ci Li, Ylva Mellbin, Johanna Krogager, Senya Polikovsky, Martin Holmberg, Nima Ghorbani, Michael J Black, Hedvig Kjellström, Silvia Zuffi, and Elin Hernlund. Scientific Data (2024).

• 3D-POP - An Automated Annotation Approach to Facilitate Markerless 2D-3D Tracking of Freely Moving Birds With Marker-Based Motion Capture. Hemal Naik, Alex Hoi Hang Chan, Junran Yang, Mathilde Delacoux, Iain D. Couzin, Fumihiro Kano, and Máté Nagy. CVPR (2023).

• Geometric deep learning enables 3D kinematic profiling across species and environments. Timothy W Dunn, Jesse D Marshall, Kyle S Severson, and et al. Nature methods (2021).

• The PAIR-R24M Dataset for Multi-animal 3D Pose Estimation. Jesse Marshall, Ugne Klibaite, amanda gellis, Diego Aldarondo, Bence Olveczky, and Timothy W Dunn. NeurIPS (2021).

RGB+Depth+MoCap

• RGBD-Dog: Predicting Canine Pose from RGBD Sensors. Sinead Kearney, Wenbin Li, Martin Parsons, Kwang In Kim, and Darren Cosker. CVPR (2020).

RGB+LiDAR

• WildPose: A Long-Range 3D Wildlife Motion Capture System. Naoya Muramatsu, Sangyun Shin, Qianyi Deng, Andrew Markham, and Amir Patel. bioRxiv (2024).

Infrared

• LoTE-Animal: A Long Time-span Dataset for Endangered Animal Behavior Understanding. Dan Liu, Jin Hou, Shaoli Huang, Jing Liu, Yuxin He, Bochuan Zheng, Jifeng Ning, and Jingdong Zhang. ICCV (2023).

3D Scanner

• VAREN: Very Accurate and Realistic Equine Network. Silvia Zuffi, Ylva Mellbin, Ci Li, Markus Hoeschle, Hedvig Kjellström, Senya Polikovsky, Elin Hernlund, and Michael J. Black. CVPR (2024).

MoCap+IMU

• Wearable inertial sensor-based limb lameness detection and pose estimation for horses. Tarik Yigit, Feng Han, Ellen Rankins, Jingang Yi, Kenneth H McKeever, and Karyn Malinowski. IEEE Transactions on Automation Science and Engineering (2022).

Todo

• Create the overall template
• Add all the papers