Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio

Praneet C. Bala, Benjamin R. Eisenreich, Seng Bum Michael Yoo, Benjamin Y. Hayden, Hyun Soo Park, Jan Zimmermann

Research output: Contribution to journalArticlepeer-review

Abstract

The rhesus macaque is an important model species in several branches of science, including neuroscience, psychology, ethology, and medicine. The utility of the macaque model would be greatly enhanced by the ability to precisely measure behavior in freely moving conditions. Existing approaches do not provide sufficient tracking. Here, we describe OpenMonkeyStudio, a deep learning-based markerless motion capture system for estimating 3D pose in freely moving macaques in large unconstrained environments. Our system makes use of 62 machine vision cameras that encircle an open 2.45 m × 2.45 m × 2.75 m enclosure. The resulting multiview image streams allow for data augmentation via 3D-reconstruction of annotated images to train a robust view-invariant deep neural network. This view invariance represents an important advance over previous markerless 2D tracking approaches, and allows fully automatic pose inference on unconstrained natural motion. We show that OpenMonkeyStudio can be used to accurately recognize actions and track social interactions.

Original languageEnglish (US)
Article number4560
JournalNature communications
Volume11
Issue number1
DOIs
StatePublished - Dec 1 2020

Bibliographical note

Funding Information:
We thank Marc Mancarella for critical initial help, Giuliana Loconte and Hannah Lee for ongoing assistance. We also thank Yasamin Jafarian and Jayant Sharma for help with developing the pipelines we used. This work was supported by an award from MNFu-tures to HSP and BYH, from the Digital Technologies Initiative to H.S.P., J.Z., and B.Y.H., from the Templeton Foundation to B.Y.H., by an R01 from NIDA (DA038615) to B.Y.H., by am NSF CAREER (1846031) to H.S.P., and by a P30 from NIDA (P30DA048742) to B.Y.H. and J.Z.

Fingerprint Dive into the research topics of 'Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio'. Together they form a unique fingerprint.

Cite this