Interactive Robotic Grasping with Attribute-Guided Disambiguation

Yang Yang, Xibai Lou, Changhyun Choi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Scopus citations


Interactive robotic grasping using natural language is one of the most fundamental tasks in human-robot interaction. However, language can be a source of ambiguity, particularly when there are ambiguous visual or linguistic contents. This paper investigates the use of object attributes in disambiguation and develops an interactive grasping system capable of effectively resolving ambiguities via dialogues. Our approach first predicts target scores and attribute scores through vision-and-language grounding. To handle ambiguous objects and commands, we propose an attribute-guided formulation of the partially observable Markov decision process (Attr-POMDP) for disambiguation. The Attr-POMDP utilizes target and attribute scores as the observation model to calculate the expected return of an attribute-based (e.g., 'what is the color of the target, red or green?') or a pointing-based (e.g., 'do you mean this one?') question. Our disambiguation module runs in real time on a real robot, and the interactive grasping system achieves a 91.43 % selection accuracy in the real-robot experiments, outperforming several baselines by large margins. Supplementary material is available at

Original languageEnglish (US)
Title of host publication2022 IEEE International Conference on Robotics and Automation, ICRA 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages7
ISBN (Electronic)9781728196817
StatePublished - 2022
Event39th IEEE International Conference on Robotics and Automation, ICRA 2022 - Philadelphia, United States
Duration: May 23 2022May 27 2022

Publication series

Name2022 International Conference on Robotics and Automation (ICRA)


Conference39th IEEE International Conference on Robotics and Automation, ICRA 2022
Country/TerritoryUnited States

Bibliographical note

Funding Information:
*This work was supported by UMII MnDRIVE Ph.D. Graduate Assistantship and MnDRIVE Initiative on Robotics, Sensors, and Advanced Manufacturing. †The authors are with University of Minnesota, Minneapolis, USA

Publisher Copyright:
© 2022 IEEE.


  • Grasping
  • Human-Centered Robotics
  • Natural Dialog for HRI


Dive into the research topics of 'Interactive Robotic Grasping with Attribute-Guided Disambiguation'. Together they form a unique fingerprint.

Cite this