A Deep Learning Approach to Grasping the Invisible

Yang Yang, Hengyue Liang, Changhyun Choi

Research output: Contribution to journalArticle

Abstract

We study an emerging problem named 'grasping the invisible' in robotic manipulation, in which a robot is tasked to grasp an initially invisible target object via a sequence of pushing and grasping actions. In this problem, pushes are needed to search for the target and rearrange cluttered objects around it to enable effective grasps. We propose to solve the problem by formulating a deep learning approach in a critic-policy format. The target-oriented motion critic, which maps both visual observations and target information to the expected future rewards of pushing and grasping motion primitives, is learned via deep Q-learning. We divide the problem into two subtasks, and two policies are proposed to tackle each of them, by combining the critic predictions and relevant domain knowledge. A Bayesian-based policy accounting for past action experience performs pushing to search for the target; once the target is found, a classifier-based policy coordinates target-oriented pushing and grasping to grasp the target in clutter. The motion critic and the classifier are trained in a self-supervised manner through robot-environment interactions. Our system achieves a 93% and 87% task success rate on each of the two subtasks in simulation and an 85% task success rate in real robot experiments on the whole problem, which outperforms several baselines by large margins. Supplementary material is available at http://sites.google.com/umn.edu/grasping-invisible.

Original languageEnglish (US)
Article number8976257
Pages (from-to)2232-2239
Number of pages8
JournalIEEE Robotics and Automation Letters
Volume5
Issue number2
DOIs
StatePublished - Apr 2020

Keywords

  • computer vision for automation
  • deep learning in robotics and automation
  • Dexterous manipulation

Fingerprint Dive into the research topics of 'A Deep Learning Approach to Grasping the Invisible'. Together they form a unique fingerprint.

  • Cite this