Look and Listen: A Multi-Sensory Pouring Network and Dataset for Granular Media from Human Demonstrations

Alexis Burns, Siyuan Xiang, Daewon Lee, Larry Jackel, Shuran Song, Volkan Isler

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Humans have the ability to pour various media, both liquid and granular, to desired ends in various containers. We do this by using multiple senses simultaneously in a constant feedback loop to complete a pouring task. Combining multiple sensing modalities, similar to humans, could aid in robotic pouring control outside of a structured or industrial setting. We present a multi-sensory pouring dataset consisting of human pouring demonstrations of various granular media, coupled with two multi-sensory networks that estimate pouring rate and pouring average height. For both pouring metrics, a combined input of audio and visual data provides a lower median error than either the audio network or visual network. The multi-sensory network achieves a median error of 6.4 mm for average height estimation and 0.06 N/s for pouring rate estimation.

Original languageEnglish (US)
Title of host publication2022 IEEE International Conference on Robotics and Automation, ICRA 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages2519-2524
Number of pages6
ISBN (Electronic)9781728196817
DOIs
StatePublished - 2022
Externally publishedYes
Event39th IEEE International Conference on Robotics and Automation, ICRA 2022 - Philadelphia, United States
Duration: May 23 2022May 27 2022

Publication series

Name2022 International Conference on Robotics and Automation (ICRA)

Conference

Conference39th IEEE International Conference on Robotics and Automation, ICRA 2022
Country/TerritoryUnited States
CityPhiladelphia
Period5/23/225/27/22

Bibliographical note

Publisher Copyright:
© 2022 IEEE.

Fingerprint

Dive into the research topics of 'Look and Listen: A Multi-Sensory Pouring Network and Dataset for Granular Media from Human Demonstrations'. Together they form a unique fingerprint.

Cite this