DiMSam: Diffusion Models as Samplers for Task and Motion Planning under
Partial Observability
- URL: http://arxiv.org/abs/2306.13196v2
- Date: Tue, 3 Oct 2023 23:52:05 GMT
- Title: DiMSam: Diffusion Models as Samplers for Task and Motion Planning under
Partial Observability
- Authors: Xiaolin Fang, Caelan Reed Garrett, Clemens Eppner, Tom\'as
Lozano-P\'erez, Leslie Pack Kaelbling, Dieter Fox
- Abstract summary: Task and Motion Planning (TAMP) approaches are effective at planning long-horizon autonomous robot manipulation.
We propose to overcome these limitations by leveraging deep generative modeling.
We show how the combination of classical TAMP, generative learning, and latent embeddings enables long-horizon constraint-based reasoning.
- Score: 50.38132214102161
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Task and Motion Planning (TAMP) approaches are effective at planning
long-horizon autonomous robot manipulation. However, it can be difficult to
apply them to domains where the environment and its dynamics are not fully
known. We propose to overcome these limitations by leveraging deep generative
modeling, specifically diffusion models, to learn constraints and samplers that
capture these difficult-to-engineer aspects of the planning model. These
learned samplers are composed and combined within a TAMP solver in order to
find action parameter values jointly that satisfy the constraints along a plan.
To tractably make predictions for unseen objects in the environment, we define
these samplers on low-dimensional learned latent embeddings of changing object
state. We evaluate our approach in an articulated object manipulation domain
and show how the combination of classical TAMP, generative learning, and latent
embeddings enables long-horizon constraint-based reasoning. We also apply the
learned sampler in the real world. More details are available at
https://sites.google.com/view/dimsam-tamp
Related papers
- Interactive Planning Using Large Language Models for Partially
Observable Robotics Tasks [54.60571399091711]
Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks.
We present an interactive planning technique for partially observable tasks using LLMs.
arXiv Detail & Related papers (2023-12-11T22:54:44Z) - Learning Extrinsic Dexterity with Parameterized Manipulation Primitives [8.7221770019454]
We learn a sequence of actions that utilize the environment to change the object's pose.
Our approach can control the object's state through exploiting interactions between the object, the gripper, and the environment.
We evaluate our approach on picking box-shaped objects of various weight, shape, and friction properties from a constrained table-top workspace.
arXiv Detail & Related papers (2023-10-26T21:28:23Z) - Robots That Ask For Help: Uncertainty Alignment for Large Language Model
Planners [85.03486419424647]
KnowNo is a framework for measuring and aligning the uncertainty of large language models.
KnowNo builds on the theory of conformal prediction to provide statistical guarantees on task completion.
arXiv Detail & Related papers (2023-07-04T21:25:12Z) - Embodied Task Planning with Large Language Models [86.63533340293361]
We propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint.
During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations.
Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin.
arXiv Detail & Related papers (2023-07-04T17:58:25Z) - Model-free Motion Planning of Autonomous Agents for Complex Tasks in
Partially Observable Environments [3.7660066212240753]
Motion planning of autonomous agents in partially known environments is a challenging problem.
This paper proposes a model-free reinforcement learning approach to address this problem.
We show that our proposed method effectively addresses environment, action, and observation uncertainties.
arXiv Detail & Related papers (2023-04-30T19:57:39Z) - Approximating Constraint Manifolds Using Generative Models for
Sampling-Based Constrained Motion Planning [8.924344714683814]
This paper presents a learning-based sampling strategy for constrained motion planning problems.
We use Conditional Variversaational Autoencoder (CVAE) and Conditional Generative Adrial Net (CGAN) to generate constraint-satisfying sample configurations.
We evaluate the efficiency of these two generative models in terms of their sampling accuracy and coverage of sampling distribution.
arXiv Detail & Related papers (2022-04-14T07:08:30Z) - SAGE: Generating Symbolic Goals for Myopic Models in Deep Reinforcement
Learning [18.37286885057802]
We propose an algorithm combining learning and planning to exploit a previously unusable class of incomplete models.
This combines the strengths of symbolic planning and neural learning approaches in a novel way that outperforms competing methods on variations of taxi world and Minecraft.
arXiv Detail & Related papers (2022-03-09T22:55:53Z) - Learning Models as Functionals of Signed-Distance Fields for
Manipulation Planning [51.74463056899926]
This work proposes an optimization-based manipulation planning framework where the objectives are learned functionals of signed-distance fields that represent objects in the scene.
We show that representing objects as signed-distance fields not only enables to learn and represent a variety of models with higher accuracy compared to point-cloud and occupancy measure representations.
arXiv Detail & Related papers (2021-10-02T12:36:58Z) - Temporal Predictive Coding For Model-Based Planning In Latent Space [80.99554006174093]
We present an information-theoretic approach that employs temporal predictive coding to encode elements in the environment that can be predicted across time.
We evaluate our model on a challenging modification of standard DMControl tasks where the background is replaced with natural videos that contain complex but irrelevant information to the planning task.
arXiv Detail & Related papers (2021-06-14T04:31:15Z) - Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a
Survey [1.2031796234206134]
Model-based reinforcement learning creates an explicit model of the environment dynamics to reduce the need for environment samples.
A challenge for deep model-based methods is to achieve high predictive power while maintaining low sample complexity.
We propose a taxonomy based on three approaches: using explicit planning on given transitions, using explicit planning on learned transitions, and end-to-end learning of both planning and transitions.
arXiv Detail & Related papers (2020-08-11T08:49:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.