Adversarial Object Rearrangement in Constrained Environments with Heterogeneous Graph Neural Networks

Xibai Lou, Houjian Yu, Ross Worobel, Yang Yang, Changhyun Choi

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    1 Scopus citations

    Abstract

    Adversarial object rearrangement in the real world (e.g., previously unseen or oversized items in kitchens and stores) could benefit from understanding task scenes, which inherently entail heterogeneous components such as current objects, goal objects, and environmental constraints. The semantic relationships among these components are distinct from each other and crucial for multi-skilled robots to perform efficiently in everyday scenarios. We propose a hierarchical robotic manipulation system that learns the underlying relationships and maximizes the collaborative power of its diverse skills (e.g., PICK-PLACE, PUSH) for rearranging adversarial objects in constrained environments. The high-level coordinator employs a heterogeneous graph neural network (HetGNN), which reasons about the current objects, goal objects, and environmental constraints; the low-level 3D Convolutional Neural Network-based actors execute the action primitives. Our approach is trained entirely in simulation, and achieved an average success rate of 87.88% and a planning cost of 12.82 in real-world experiments, surpassing all baseline methods. Supplementary material is available at https://sites.google.com/umn.edu/versatile-rearrangement.

    Original languageEnglish (US)
    Title of host publication2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2023
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Pages1008-1015
    Number of pages8
    ISBN (Electronic)9781665491907
    DOIs
    StatePublished - 2023
    Event2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2023 - Detroit, United States
    Duration: Oct 1 2023Oct 5 2023

    Publication series

    NameIEEE International Conference on Intelligent Robots and Systems
    ISSN (Print)2153-0858
    ISSN (Electronic)2153-0866

    Conference

    Conference2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2023
    Country/TerritoryUnited States
    CityDetroit
    Period10/1/2310/5/23

    Bibliographical note

    Publisher Copyright:
    © 2023 IEEE.

    Keywords

    • Deep Learning in Grasping and Manipulation
    • Perception for Grasping and Manipulation

    Fingerprint

    Dive into the research topics of 'Adversarial Object Rearrangement in Constrained Environments with Heterogeneous Graph Neural Networks'. Together they form a unique fingerprint.

    Cite this