Self-supervised Secondary Landmark Detection via 3D Representation Learning

Praneet Bala, Jan Zimmermann, Hyun Soo Park, Benjamin Y. Hayden

Research output: Contribution to journalArticlepeer-review

3 Scopus citations


Recent technological developments have lead to great advances in the computerized tracking of joints and other landmarks in moving animals, including humans. Such tracking promises important advances in biology and biomedicine. Modern tracking models depend critically on labor-intensive annotated datasets of primary landmarks by non-expert humans. However, such annotation approaches can be costly and impractical for secondary landmarks, that is, ones that reflect fine-grained geometry of animals, and that are often specific to customized behavioral tasks. Due to visual and geometric ambiguity, non-experts are often not qualified for secondary landmark annotation, which can require anatomical and zoological knowledge. These barriers significantly impede downstream behavioral studies because the learned tracking models exhibit limited generalizability. We hypothesize that there exists a shared representation between the primary and secondary landmarks because the range of motion of the secondary landmarks can be approximately spanned by that of the primary landmarks. We present a method to learn this spatial relationship of the primary and secondary landmarks in three dimensional space, which can, in turn, self-supervise the secondary landmark detector. This 3D representation learning is generic, and can therefore be applied to various multiview settings across diverse organisms, including macaques, flies, and humans.

Original languageEnglish (US)
Pages (from-to)1980-1994
Number of pages15
JournalInternational Journal of Computer Vision
Issue number8
StatePublished - Aug 2023

Bibliographical note

Publisher Copyright:
© 2023, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.


  • Contrastive learning
  • Human and non-human dataset
  • Landmark detection
  • Self-supervised learning
  • Shared representations


Dive into the research topics of 'Self-supervised Secondary Landmark Detection via 3D Representation Learning'. Together they form a unique fingerprint.

Cite this