TY - JOUR
T1 - Lightning Pose
T2 - improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools
AU - The International Brain Laboratory
AU - Biderman, Dan
AU - Whiteway, Matthew R.
AU - Hurwitz, Cole
AU - Greenspan, Nicholas
AU - Lee, Robert S.
AU - Vishnubhotla, Ankit
AU - Warren, Richard
AU - Pedraja, Federico
AU - Noone, Dillon
AU - Schartner, Michael M.
AU - Huntenburg, Julia M.
AU - Khanal, Anup
AU - Meijer, Guido T.
AU - Noel, Jean Paul
AU - Pan-Vazquez, Alejandro
AU - Socha, Karolina Z.
AU - Urai, Anne E.
AU - Zhang, Yizi
AU - Zador, Anthony M.
AU - Yu, Han
AU - Xu, Zekai
AU - Wool, Lauren E.
AU - Witten, Ilana
AU - Winter, Olivier
AU - Windolf, Charles
AU - Whiteway, Matthew R.
AU - West, Steven J.
AU - Wells, Miles J.
AU - Varol, Erdem
AU - Urai, Anne E.
AU - Tessereau, Charline
AU - Taheri, Marsa
AU - Svoboda, Karel
AU - Steinmetz, Nicholas A.
AU - Soitu, Cristian
AU - Soares, Carolina
AU - Shi, Yanliang
AU - Schartner, Michael M.
AU - Schaeffer, Rylan
AU - Saniee, Kamron
AU - Roy, Nicholas A.
AU - Roth, Noam
AU - Rossant, Cyrille
AU - Rau, Florian
AU - Pouget, Alexandre
AU - Pillow, Jonathan W.
AU - Picard, Samuel
AU - Pezzotta, Alberto
AU - Paninski, Liam
AU - Oloomi, Farideh
N1 - Publisher Copyright:
© The Author(s), under exclusive licence to Springer Nature America, Inc. 2024.
PY - 2024/7
Y1 - 2024/7
N2 - Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce ‘Lightning Pose’, an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.
AB - Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce ‘Lightning Pose’, an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.
UR - http://www.scopus.com/inward/record.url?scp=85198569858&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85198569858&partnerID=8YFLogxK
U2 - 10.1038/s41592-024-02319-1
DO - 10.1038/s41592-024-02319-1
M3 - Article
C2 - 38918605
AN - SCOPUS:85198569858
SN - 1548-7091
VL - 21
SP - 1316
EP - 1328
JO - Nature Methods
JF - Nature Methods
IS - 7
ER -