First-person action-object detection with EgoNet

Gedas Bertasius, Hyun Soo Park, Stella X. Yu, Jianbo Shi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Scopus citations

Abstract

Unlike traditional third-person cameras mounted on robots, a first-person camera, captures a person's visual sensorimotor object interactions from up close. In this paper, we study the tight interplay between our momentary visual attention and motor action with objects from a first-person camera. We propose a concept of action-objects-the objects that capture person's conscious visual (watching a TV) or tactile (taking a cup) interactions. Action-objects may be task-dependent but since many tasks share common person-object spatial configurations, action-objects exhibit a characteristic 3D spatial distance and orientation with respect to the person. We design a predictive model that detects action-objects using EgoNet, a joint two-stream network that holistically integrates visual appearance (RGB) and 3D spatial layout (depth and height) cues to predict per-pixel likelihood of action-objects. Our network also incorporates a first-person coordinate embedding, which is designed to learn a spatial distribution of the actionobjects in the first-person data. We demonstrate EgoNet's predictive power, by showing that it consistently outperforms previous baseline approaches. Furthermore, EgoNet also exhibits a strong generalization ability, i.e., it predicts semantically meaningful objects in novel first-person datasets. Our method's ability to effectively detect action-objects could be used to improve robots' understanding of human-object interactions.

Original languageEnglish (US)
Title of host publicationRobotics
Subtitle of host publicationScience and Systems XIII, RSS 2017
EditorsNancy Amato, Siddhartha Srinivasa, Nora Ayanian, Scott Kuindersma
PublisherMIT Press Journals
ISBN (Electronic)9780992374730
DOIs
StatePublished - 2017
Externally publishedYes
Event2017 Robotics: Science and Systems, RSS 2017 - Cambridge, United States
Duration: Jul 12 2017Jul 16 2017

Publication series

NameRobotics: Science and Systems
Volume13
ISSN (Electronic)2330-765X

Other

Other2017 Robotics: Science and Systems, RSS 2017
CountryUnited States
CityCambridge
Period7/12/177/16/17

Fingerprint Dive into the research topics of 'First-person action-object detection with EgoNet'. Together they form a unique fingerprint.

Cite this