Ego4D: Around the World in 3,000 Hours of Egocentric Video

Kristen Grauman; Andrew Westbury; Eugene Byrne; Zachary Chavis; Antonino Furnari; Rohit Girdhar; Jackson Hamburger; Hao Jiang; Miao Liu; Xingyu Liu; Miguel Martin; Tushar Nagarajan; Ilija Radosavovic; Santhosh Kumar Ramakrishnan; Fiona Ryan; Jayant Sharma; Michael Wray; Mengmeng Xu; Eric Zhongcong Xu; Chen Zhao; Siddhant Bansal; Dhruv Batra; Vincent Cartillier; Sean Crane; Tien Do; Morrie Doulaty; Akshay Erapalli; Christoph Feichtenhofer; Adriano Fragomeni; Qichen Fu; Abrham Gebreselasie; Cristina Gonzalez; James Hillis; Xuhua Huang; Yifei Huang; Wenqi Jia; Weslie Khoo; Jachym Kolar; Satwik Kottur; Anurag Kumar; Federico Landini; Chao Li; Yanghao Li; Zhenqiang Li; Karttikeya Mangalam; Raghava Modhugu; Jonathan Munro; Tullie Murrell; Takumi Nishiyasu; Will Price; Paola Ruiz Puentes; Merey Ramazanova; Leda Sari; Kiran Somasundaram; Audrey Southerland; Yusuke Sugano; Ruijie Tao; Minh Vo; Yuchen Wang; Xindi Wu; Takuma Yagi; Ziwei Zhao; Yunyi Zhu; Pablo Arbelaez; David Crandall; Dima Damen; Giovanni Maria Farinella; Christian Fuegen; Bernard Ghanem; Vamsi Krishna Ithapu; C. V. Jawahar; Hanbyul Joo; Kris Kitani; Haizhou Li; Richard Newcombe; Aude Oliva; Hyun Soo Park; James M. Rehg; Yoichi Sato; Jianbo Shi; Mike Zheng Shou; Antonio Torralba; Lorenzo Torresani; Mingfei Yan; Jitendra Malik

doi:10.1109/CVPR52688.2022.01842

Ego4D: Around the World in 3,000 Hours of Egocentric Video

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen ZhaoSiddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, Jitendra Malik

Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

546 Scopus citations

Abstract

We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of dailylife activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations and 9 different countries. The approach to collection is designed to uphold rigorous privacy and ethics standards, with consenting participants and robust de-identification procedures where relevant. Ego4D dramatically expands the volume of diverse egocentric video footage publicly available to the research community. Portions of the video are accompanied by audio, 3D meshes of the environment, eye gaze, stereo, and/or synchronized videos from multiple egocentric cameras at the same event. Furthermore, we present a host of new benchmark challenges centered around understanding the first-person visual experience in the past (querying an episodic memory), present (analyzing hand-object manipulation, audio-visual conversation, and social interactions), and future (forecasting activities). By publicly sharing this massive annotated dataset and benchmark suite, we aim to push the frontier of first-person perception. Project page: https://ego4d-data.org/

Original language	English (US)
Title of host publication	Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
Publisher	IEEE Computer Society
Pages	18973-18990
Number of pages	18
ISBN (Electronic)	9781665469463
DOIs	https://doi.org/10.1109/CVPR52688.2022.01842
State	Published - 2022
Event	2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 - New Orleans, United States Duration: Jun 19 2022 → Jun 24 2022

Publication series

Name	Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Volume	2022-June
ISSN (Print)	1063-6919

Conference

Conference	2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
Country/Territory	United States
City	New Orleans
Period	6/19/22 → 6/24/22

Bibliographical note

Funding Information:
UNICT is supported by MIUR AIM - Attrazione e MobilitaIn-ternazionale Linea 1 - AIM1893589 - CUP E64118002540007. Bristol is supported by UKRIEngineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EP-SRC Fellowship UMPIRE (EP/T004991/1). KAUST is supported by the KAUST Office of Sponsored Research through the Visual Computing Center (VCC) funding. National University of Singapore is supported by Mike Shou’s Start-Up Grant. Georgia Tech is supported in part by NSF 2033413 and NIH R01MH114999.

Publisher Copyright:
© 2022 IEEE.

Keywords

Datasets and evaluation
Video analysis and understanding

Publisher link

10.1109/CVPR52688.2022.01842

Find It

Find It at U of M Libraries

Cite this

Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., Martin, M., Nagarajan, T., Radosavovic, I., Ramakrishnan, S. K., Ryan, F., Sharma, J., Wray, M., Xu, M., Xu, E. Z., ... Malik, J. (2022). Ego4D: Around the World in 3,000 Hours of Egocentric Video. In Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 (pp. 18973-18990). (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 2022-June). IEEE Computer Society. https://doi.org/10.1109/CVPR52688.2022.01842

Ego4D: Around the World in 3,000 Hours of Egocentric Video. / Grauman, Kristen; Westbury, Andrew; Byrne, Eugene et al.
Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. IEEE Computer Society, 2022. p. 18973-18990 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 2022-June).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Grauman, K, Westbury, A, Byrne, E, Chavis, Z, Furnari, A, Girdhar, R, Hamburger, J, Jiang, H, Liu, M, Liu, X, Martin, M, Nagarajan, T, Radosavovic, I, Ramakrishnan, SK, Ryan, F, Sharma, J, Wray, M, Xu, M, Xu, EZ, Zhao, C, Bansal, S, Batra, D, Cartillier, V, Crane, S, Do, T, Doulaty, M, Erapalli, A, Feichtenhofer, C, Fragomeni, A, Fu, Q, Gebreselasie, A, Gonzalez, C, Hillis, J, Huang, X, Huang, Y, Jia, W, Khoo, W, Kolar, J, Kottur, S, Kumar, A, Landini, F, Li, C, Li, Y, Li, Z, Mangalam, K, Modhugu, R, Munro, J, Murrell, T, Nishiyasu, T, Price, W, Puentes, PR, Ramazanova, M, Sari, L, Somasundaram, K, Southerland, A, Sugano, Y, Tao, R, Vo, M, Wang, Y, Wu, X, Yagi, T, Zhao, Z, Zhu, Y, Arbelaez, P, Crandall, D, Damen, D, Farinella, GM, Fuegen, C, Ghanem, B, Ithapu, VK, Jawahar, CV, Joo, H, Kitani, K, Li, H, Newcombe, R, Oliva, A, Park, HS, Rehg, JM, Sato, Y, Shi, J, Shou, MZ, Torralba, A, Torresani, L, Yan, M & Malik, J 2022, Ego4D: Around the World in 3,000 Hours of Egocentric Video. in Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2022-June, IEEE Computer Society, pp. 18973-18990, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, United States, 6/19/22. https://doi.org/10.1109/CVPR52688.2022.01842

Grauman K, Westbury A, Byrne E, Chavis Z, Furnari A, Girdhar R et al. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. IEEE Computer Society. 2022. p. 18973-18990. (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition). doi: 10.1109/CVPR52688.2022.01842

@inproceedings{c42c3f64ff7f40aea136308ad767c078,

title = "Ego4D: Around the World in 3,000 Hours of Egocentric Video",

abstract = "We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of dailylife activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations and 9 different countries. The approach to collection is designed to uphold rigorous privacy and ethics standards, with consenting participants and robust de-identification procedures where relevant. Ego4D dramatically expands the volume of diverse egocentric video footage publicly available to the research community. Portions of the video are accompanied by audio, 3D meshes of the environment, eye gaze, stereo, and/or synchronized videos from multiple egocentric cameras at the same event. Furthermore, we present a host of new benchmark challenges centered around understanding the first-person visual experience in the past (querying an episodic memory), present (analyzing hand-object manipulation, audio-visual conversation, and social interactions), and future (forecasting activities). By publicly sharing this massive annotated dataset and benchmark suite, we aim to push the frontier of first-person perception. Project page: https://ego4d-data.org/",

keywords = "Datasets and evaluation, Video analysis and understanding",

author = "Kristen Grauman and Andrew Westbury and Eugene Byrne and Zachary Chavis and Antonino Furnari and Rohit Girdhar and Jackson Hamburger and Hao Jiang and Miao Liu and Xingyu Liu and Miguel Martin and Tushar Nagarajan and Ilija Radosavovic and Ramakrishnan, \{Santhosh Kumar\} and Fiona Ryan and Jayant Sharma and Michael Wray and Mengmeng Xu and Xu, \{Eric Zhongcong\} and Chen Zhao and Siddhant Bansal and Dhruv Batra and Vincent Cartillier and Sean Crane and Tien Do and Morrie Doulaty and Akshay Erapalli and Christoph Feichtenhofer and Adriano Fragomeni and Qichen Fu and Abrham Gebreselasie and Cristina Gonzalez and James Hillis and Xuhua Huang and Yifei Huang and Wenqi Jia and Weslie Khoo and Jachym Kolar and Satwik Kottur and Anurag Kumar and Federico Landini and Chao Li and Yanghao Li and Zhenqiang Li and Karttikeya Mangalam and Raghava Modhugu and Jonathan Munro and Tullie Murrell and Takumi Nishiyasu and Will Price and Puentes, \{Paola Ruiz\} and Merey Ramazanova and Leda Sari and Kiran Somasundaram and Audrey Southerland and Yusuke Sugano and Ruijie Tao and Minh Vo and Yuchen Wang and Xindi Wu and Takuma Yagi and Ziwei Zhao and Yunyi Zhu and Pablo Arbelaez and David Crandall and Dima Damen and Farinella, \{Giovanni Maria\} and Christian Fuegen and Bernard Ghanem and Ithapu, \{Vamsi Krishna\} and Jawahar, \{C. V.\} and Hanbyul Joo and Kris Kitani and Haizhou Li and Richard Newcombe and Aude Oliva and Park, \{Hyun Soo\} and Rehg, \{James M.\} and Yoichi Sato and Jianbo Shi and Shou, \{Mike Zheng\} and Antonio Torralba and Lorenzo Torresani and Mingfei Yan and Jitendra Malik",

note = "Funding Information: UNICT is supported by MIUR AIM - Attrazione e MobilitaIn-ternazionale Linea 1 - AIM1893589 - CUP E64118002540007. Bristol is supported by UKRIEngineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EP-SRC Fellowship UMPIRE (EP/T004991/1). KAUST is supported by the KAUST Office of Sponsored Research through the Visual Computing Center (VCC) funding. National University of Singapore is supported by Mike Shou{\textquoteright}s Start-Up Grant. Georgia Tech is supported in part by NSF 2033413 and NIH R01MH114999. Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 ; Conference date: 19-06-2022 Through 24-06-2022",

year = "2022",

doi = "10.1109/CVPR52688.2022.01842",

language = "English (US)",

series = "Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition",

publisher = "IEEE Computer Society",

pages = "18973--18990",

booktitle = "Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022",

}

TY - GEN

T1 - Ego4D

T2 - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022

AU - Grauman, Kristen

AU - Westbury, Andrew

AU - Byrne, Eugene

AU - Chavis, Zachary

AU - Furnari, Antonino

AU - Girdhar, Rohit

AU - Hamburger, Jackson

AU - Jiang, Hao

AU - Liu, Miao

AU - Liu, Xingyu

AU - Martin, Miguel

AU - Nagarajan, Tushar

AU - Radosavovic, Ilija

AU - Ramakrishnan, Santhosh Kumar

AU - Ryan, Fiona

AU - Sharma, Jayant

AU - Wray, Michael

AU - Xu, Mengmeng

AU - Xu, Eric Zhongcong

AU - Zhao, Chen

AU - Bansal, Siddhant

AU - Batra, Dhruv

AU - Cartillier, Vincent

AU - Crane, Sean

AU - Do, Tien

AU - Doulaty, Morrie

AU - Erapalli, Akshay

AU - Feichtenhofer, Christoph

AU - Fragomeni, Adriano

AU - Fu, Qichen

AU - Gebreselasie, Abrham

AU - Gonzalez, Cristina

AU - Hillis, James

AU - Huang, Xuhua

AU - Huang, Yifei

AU - Jia, Wenqi

AU - Khoo, Weslie

AU - Kolar, Jachym

AU - Kottur, Satwik

AU - Kumar, Anurag

AU - Landini, Federico

AU - Li, Chao

AU - Li, Yanghao

AU - Li, Zhenqiang

AU - Mangalam, Karttikeya

AU - Modhugu, Raghava

AU - Munro, Jonathan

AU - Murrell, Tullie

AU - Nishiyasu, Takumi

AU - Price, Will

AU - Puentes, Paola Ruiz

AU - Ramazanova, Merey

AU - Sari, Leda

AU - Somasundaram, Kiran

AU - Southerland, Audrey

AU - Sugano, Yusuke

AU - Tao, Ruijie

AU - Vo, Minh

AU - Wang, Yuchen

AU - Wu, Xindi

AU - Yagi, Takuma

AU - Zhao, Ziwei

AU - Zhu, Yunyi

AU - Arbelaez, Pablo

AU - Crandall, David

AU - Damen, Dima

AU - Farinella, Giovanni Maria

AU - Fuegen, Christian

AU - Ghanem, Bernard

AU - Ithapu, Vamsi Krishna

AU - Jawahar, C. V.

AU - Joo, Hanbyul

AU - Kitani, Kris

AU - Li, Haizhou

AU - Newcombe, Richard

AU - Oliva, Aude

AU - Park, Hyun Soo

AU - Rehg, James M.

AU - Sato, Yoichi

AU - Shi, Jianbo

AU - Shou, Mike Zheng

AU - Torralba, Antonio

AU - Torresani, Lorenzo

AU - Yan, Mingfei

AU - Malik, Jitendra

N1 - Funding Information: UNICT is supported by MIUR AIM - Attrazione e MobilitaIn-ternazionale Linea 1 - AIM1893589 - CUP E64118002540007. Bristol is supported by UKRIEngineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EP-SRC Fellowship UMPIRE (EP/T004991/1). KAUST is supported by the KAUST Office of Sponsored Research through the Visual Computing Center (VCC) funding. National University of Singapore is supported by Mike Shou’s Start-Up Grant. Georgia Tech is supported in part by NSF 2033413 and NIH R01MH114999. Publisher Copyright: © 2022 IEEE.

PY - 2022

Y1 - 2022

N2 - We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of dailylife activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations and 9 different countries. The approach to collection is designed to uphold rigorous privacy and ethics standards, with consenting participants and robust de-identification procedures where relevant. Ego4D dramatically expands the volume of diverse egocentric video footage publicly available to the research community. Portions of the video are accompanied by audio, 3D meshes of the environment, eye gaze, stereo, and/or synchronized videos from multiple egocentric cameras at the same event. Furthermore, we present a host of new benchmark challenges centered around understanding the first-person visual experience in the past (querying an episodic memory), present (analyzing hand-object manipulation, audio-visual conversation, and social interactions), and future (forecasting activities). By publicly sharing this massive annotated dataset and benchmark suite, we aim to push the frontier of first-person perception. Project page: https://ego4d-data.org/

AB - We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of dailylife activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations and 9 different countries. The approach to collection is designed to uphold rigorous privacy and ethics standards, with consenting participants and robust de-identification procedures where relevant. Ego4D dramatically expands the volume of diverse egocentric video footage publicly available to the research community. Portions of the video are accompanied by audio, 3D meshes of the environment, eye gaze, stereo, and/or synchronized videos from multiple egocentric cameras at the same event. Furthermore, we present a host of new benchmark challenges centered around understanding the first-person visual experience in the past (querying an episodic memory), present (analyzing hand-object manipulation, audio-visual conversation, and social interactions), and future (forecasting activities). By publicly sharing this massive annotated dataset and benchmark suite, we aim to push the frontier of first-person perception. Project page: https://ego4d-data.org/

KW - Datasets and evaluation

KW - Video analysis and understanding

UR - http://www.scopus.com/inward/record.url?scp=85141776718&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85141776718&partnerID=8YFLogxK

U2 - 10.1109/CVPR52688.2022.01842

DO - 10.1109/CVPR52688.2022.01842

M3 - Conference contribution

AN - SCOPUS:85141776718

T3 - Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

SP - 18973

EP - 18990

BT - Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022

PB - IEEE Computer Society

Y2 - 19 June 2022 through 24 June 2022

ER -

Ego4D: Around the World in 3,000 Hours of Egocentric Video

Abstract

Publication series

Conference

Bibliographical note

Keywords

Publisher link

Find It

Other files and links

Fingerprint

Cite this