"Set It Up!": Functional Object Arrangement with Compositional Generative Models

• URL: http://arxiv.org/abs/2405.11928v2
• Date: Wed, 16 Oct 2024 04:35:31 GMT
• Title: "Set It Up!": Functional Object Arrangement with Compositional Generative Models
• Authors: Yiqing Xu, Jiayuan Mao, Yilun Du, Tomas Lozáno-Pérez, Leslie Pack Kaebling, David Hsu,
• Abstract summary: We introduce a framework, SetItUp, for learning to interpret under-specified instructions. We validate our framework on a dataset comprising study desks, dining tables, and coffee tables.
• Score: 48.205899984212074
• License:
• Abstract: This paper studies the challenge of developing robots capable of understanding under-specified instructions for creating functional object arrangements, such as "set up a dining table for two"; previous arrangement approaches have focused on much more explicit instructions, such as "put object A on the table." We introduce a framework, SetItUp, for learning to interpret under-specified instructions. SetItUp takes a small number of training examples and a human-crafted program sketch to uncover arrangement rules for specific scene types. By leveraging an intermediate graph-like representation of abstract spatial relationships among objects, SetItUp decomposes the arrangement problem into two subproblems: i) learning the arrangement patterns from limited data and ii) grounding these abstract relationships into object poses. SetItUp leverages large language models (LLMs) to propose the abstract spatial relationships among objects in novel scenes as the constraints to be satisfied; then, it composes a library of diffusion models associated with these abstract relationships to find object poses that satisfy the constraints. We validate our framework on a dataset comprising study desks, dining tables, and coffee tables, with the results showing superior performance in generating physically plausible, functional, and aesthetically pleasing object arrangements compared to existing models.

Related papers

• Constrained Layout Generation with Factor Graphs [21.07236104467961]
  We introduce a factor graph based approach with four latent variable nodes for each room, and a factor node for each constraint. The factor nodes represent dependencies among the variables to which they are connected, effectively capturing constraints that are potentially of a higher order. Our approach is simple and generates layouts faithful to the user requirements, demonstrated by a large improvement in IOU scores over existing methods.
  arXiv  Detail & Related papers  (2024-03-30T14:58:40Z)
• Multi-Object Graph Affordance Network: Goal-Oriented Planning through Learned Compound Object Affordances [1.9336815376402723]
  The Multi-Object Graph Affordance Network models complex compound object affordances by learning the outcomes of robot actions that facilitate interactions between an object and a compound. We show that our system successfully modeled the affordances of compound objects that include concave and convex objects, in both simulated and real-world environments.
  arXiv  Detail & Related papers  (2023-09-19T08:40:46Z)
• Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement [75.9289887536165]
  We present a hierarchical abstraction approach to uncover underlying entities. We show how to learn a correspondence between intervening on states of entities in the agent's model and acting on objects in the environment. We use this correspondence to develop a method for control that generalizes to different numbers and configurations of objects.
  arXiv  Detail & Related papers  (2023-03-20T18:19:36Z)
• ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select [53.071352033539526]
  We study the problem of extracting N-ary relations from scientific articles. Our proposed method ReSel decomposes this task into a two-stage procedure. Our experiments on three scientific information extraction datasets show that ReSel outperforms state-of-the-art baselines significantly.
  arXiv  Detail & Related papers  (2022-10-26T02:28:02Z)
• Learning to Compose Visual Relations [100.45138490076866]
  We propose to represent each relation as an unnormalized density (an energy-based model) We show that such a factorized decomposition allows the model to both generate and edit scenes with multiple sets of relations more faithfully.
  arXiv  Detail & Related papers  (2021-11-17T18:51:29Z)
• Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning [51.74463056899926]
  This work proposes an optimization-based manipulation planning framework where the objectives are learned functionals of signed-distance fields that represent objects in the scene. We show that representing objects as signed-distance fields not only enables to learn and represent a variety of models with higher accuracy compared to point-cloud and occupancy measure representations.
  arXiv  Detail & Related papers  (2021-10-02T12:36:58Z)
• Predicting Stable Configurations for Semantic Placement of Novel Objects [37.18437299513799]
  Our goal is to enable robots to repose previously unseen objects according to learned semantic relationships in novel environments. We build our models and training from the ground up to be tightly integrated with our proposed planning algorithm for semantic placement of unknown objects. Our approach enables motion planning for semantic rearrangement of unknown objects in scenes with varying geometry from only RGB-D sensing.
  arXiv  Detail & Related papers  (2021-08-26T23:05:05Z)
• Efficient State Abstraction using Object-centered Predicates for Manipulation Planning [86.24148040040885]
  We propose an object-centered representation that permits characterizing a much wider set of possible changes in configuration spaces. Based on this representation, we define universal planning operators for picking and placing actions that permits generating plans with geometric and force consistency.
  arXiv  Detail & Related papers  (2020-07-16T10:52:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.