IOSG: Image-Driven Object Searching and Grasping

Houjian Yu, Xibai Lou, Yang Yang, Changhyun Choi

    Research output: Chapter in Book/Report/Conference proceedingConference contribution


    When robots retrieve specific objects from cluttered scenes, such as home and warehouse environments, the target objects are often partially occluded or completely hidden. Robots are thus required to search, identify a target object, and successfully grasp it. Preceding works have relied on pre-trained object recognition or segmentation models to find the target object. However, such methods require laborious manual annotations to train the models and even fail to find novel target objects. In this paper, we propose an Image-driven Object Searching and Grasping (IOSG) approach where a robot is provided with the reference image of a novel target object and tasked to find and retrieve it. We design a Target Similarity Network that generates a probability map to infer the location of the novel target. IOSG learns a hierarchical policy; the high-level policy predicts the subtask type, whereas the low-level policies, explorer and coordinator, generate effective push and grasp actions. The explorer is responsible for searching the target object when it is hidden or occluded by other objects. Once the target object is found, the coordinator conducts target-oriented pushing and grasping to retrieve the target from the clutter. The proposed pipeline is trained with full self-supervision in simulation and applied to a real environment. Our model achieves a 96.0% and 94.5% task success rate on coordination and exploration tasks in simulation respectively, and 85.0% success rate on a real robot for the search-and-grasp task. Please refer to our project page for more information:

    Original languageEnglish (US)
    Title of host publication2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2023
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Number of pages8
    ISBN (Electronic)9781665491907
    StatePublished - 2023
    Event2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2023 - Detroit, United States
    Duration: Oct 1 2023Oct 5 2023

    Publication series

    NameIEEE International Conference on Intelligent Robots and Systems
    ISSN (Print)2153-0858
    ISSN (Electronic)2153-0866


    Conference2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2023
    Country/TerritoryUnited States

    Bibliographical note

    Publisher Copyright:
    © 2023 IEEE.

    Cite this