Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning

• URL: http://arxiv.org/abs/2110.00792v1
• Date: Sat, 2 Oct 2021 12:36:58 GMT
• Title: Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning
• Authors: Danny Driess, Jung-Su Ha, Marc Toussaint, Russ Tedrake
• Abstract summary: This work proposes an optimization-based manipulation planning framework where the objectives are learned functionals of signed-distance fields that represent objects in the scene. We show that representing objects as signed-distance fields not only enables to learn and represent a variety of models with higher accuracy compared to point-cloud and occupancy measure representations.
• Score: 51.74463056899926
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work proposes an optimization-based manipulation planning framework where the objectives are learned functionals of signed-distance fields that represent objects in the scene. Most manipulation planning approaches rely on analytical models and carefully chosen abstractions/state-spaces to be effective. A central question is how models can be obtained from data that are not primarily accurate in their predictions, but, more importantly, enable efficient reasoning within a planning framework, while at the same time being closely coupled to perception spaces. We show that representing objects as signed-distance fields not only enables to learn and represent a variety of models with higher accuracy compared to point-cloud and occupancy measure representations, but also that SDF-based models are suitable for optimization-based planning. To demonstrate the versatility of our approach, we learn both kinematic and dynamic models to solve tasks that involve hanging mugs on hooks and pushing objects on a table. We can unify these quite different tasks within one framework, since SDFs are the common object representation. Video: https://youtu.be/ga8Wlkss7co

Related papers

• Keypoint Abstraction using Large Models for Object-Relative Imitation Learning [78.92043196054071]
  Generalization to novel object configurations and instances across diverse tasks and environments is a critical challenge in robotics. Keypoint-based representations have been proven effective as a succinct representation for essential object capturing features. We propose KALM, a framework that leverages large pre-trained vision-language models to automatically generate task-relevant and cross-instance consistent keypoints.
  arXiv  Detail & Related papers  (2024-10-30T17:37:31Z)
• Compositional Foundation Models for Hierarchical Planning [52.18904315515153]
  We propose a foundation model which leverages expert foundation model trained on language, vision and action data individually together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos.
  arXiv  Detail & Related papers  (2023-09-15T17:44:05Z)
• An Object SLAM Framework for Association, Mapping, and High-Level Tasks [12.62957558651032]
  We present a comprehensive object SLAM framework that focuses on object-based perception and object-oriented robot tasks. A range of public datasets and real-world results have been used to evaluate the proposed object SLAM framework for its efficient performance.
  arXiv  Detail & Related papers  (2023-05-12T08:10:14Z)
• LLM2Loss: Leveraging Language Models for Explainable Model Diagnostics [5.33024001730262]
  We propose an approach that can provide semantic insights into a model's patterns of failures and biases. We show that an ensemble of such lightweight models can be used to generate insights on the performance of the black-box model.
  arXiv  Detail & Related papers  (2023-05-04T23:54:37Z)
• KINet: Unsupervised Forward Models for Robotic Pushing Manipulation [8.572983995175909]
  We introduce KINet -- an unsupervised framework to reason about object interactions based on a keypoint representation. Our model learns to associate objects with keypoint coordinates and discovers a graph representation of the system. By learning to perform physical reasoning in the keypoint space, our model automatically generalizes to scenarios with a different number of objects.
  arXiv  Detail & Related papers  (2022-02-18T03:32:08Z)
• Model-Based Visual Planning with Self-Supervised Functional Distances [104.83979811803466]
  We present a self-supervised method for model-based visual goal reaching. Our approach learns entirely using offline, unlabeled data. We find that this approach substantially outperforms both model-free and model-based prior methods.
  arXiv  Detail & Related papers  (2020-12-30T23:59:09Z)
• Learning Long-term Visual Dynamics with Region Proposal Interaction Networks [75.06423516419862]
  We build object representations that can capture inter-object and object-environment interactions over a long-range. Thanks to the simple yet effective object representation, our approach outperforms prior methods by a significant margin.
  arXiv  Detail & Related papers  (2020-08-05T17:48:00Z)
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation [83.16948429592621]
  We propose a new learning framework that jointly optimize both the visual representation model and the dynamics model. We show substantial improvements over standard model-based learning techniques across our rope and cloth manipulation suite.
  arXiv  Detail & Related papers  (2020-03-11T17:55:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.