Diffusion Meets Options: Hierarchical Generative Skill Composition for Temporally-Extended Tasks

• URL: http://arxiv.org/abs/2410.02389v1
• Date: Thu, 3 Oct 2024 11:10:37 GMT
• Title: Diffusion Meets Options: Hierarchical Generative Skill Composition for Temporally-Extended Tasks
• Authors: Zeyu Feng, Hao Luan, Kevin Yuchen Ma, Harold Soh,
• Abstract summary: We propose a data-driven hierarchical framework that generates and updates plans based on instruction specified by linear temporal logic (LTL) Our method decomposes temporal tasks into chain of options with hierarchical reinforcement learning from offline non-expert datasets. We devise a determinantal-guided posterior sampling technique during batch generation, which improves the speed and diversity of diffusion generated options.
• Score: 12.239868705130178
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Safe and successful deployment of robots requires not only the ability to generate complex plans but also the capacity to frequently replan and correct execution errors. This paper addresses the challenge of long-horizon trajectory planning under temporally extended objectives in a receding horizon manner. To this end, we propose DOPPLER, a data-driven hierarchical framework that generates and updates plans based on instruction specified by linear temporal logic (LTL). Our method decomposes temporal tasks into chain of options with hierarchical reinforcement learning from offline non-expert datasets. It leverages diffusion models to generate options with low-level actions. We devise a determinantal-guided posterior sampling technique during batch generation, which improves the speed and diversity of diffusion generated options, leading to more efficient querying. Experiments on robot navigation and manipulation tasks demonstrate that DOPPLER can generate sequences of trajectories that progressively satisfy the specified formulae for obstacle avoidance and sequential visitation. Demonstration videos are available online at: https://philiptheother.github.io/doppler/.

Related papers

• Task-agnostic Pre-training and Task-guided Fine-tuning for Versatile Diffusion Planner [12.360598915420255]
  We propose textbfSODP, a two-stage framework that learns a textbfDiffusion textbfPlanner, which is generalizable for various downstream tasks. In the pre-training stage, we train a foundation diffusion planner that extracts general planning capabilities by modeling the versatile distribution of multi-task trajectories. Then for downstream tasks, we adopt RL-based fine-tuning with task-specific rewards to fast refine the diffusion planner.
  arXiv  Detail & Related papers  (2024-09-30T05:05:37Z)
• Directed Exploration in Reinforcement Learning from Linear Temporal Logic [59.707408697394534]
  Linear temporal logic (LTL) is a powerful language for task specification in reinforcement learning. We show that the synthesized reward signal remains fundamentally sparse, making exploration challenging. We show how better exploration can be achieved by further leveraging the specification and casting its corresponding Limit Deterministic B"uchi Automaton (LDBA) as a Markov reward process.
  arXiv  Detail & Related papers  (2024-08-18T14:25:44Z)
• PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer [47.924941959320996]
  We propose a hierarchical planner designed for offline RL called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level, named D-Conductor, which guides the low-level policy through sub-goals. At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals.
  arXiv  Detail & Related papers  (2024-06-10T20:59:53Z)
• LTLDoG: Satisfying Temporally-Extended Symbolic Constraints for Safe Diffusion-based Planning [12.839846486863308]
  In this work, we focus on generating long-horizon trajectories that adhere to novel static and temporally-extended constraints/instructions at test time. We propose a data-driven diffusion-based framework, finiteDoG, that modifies the inference steps of the reverse process given an instruction specified using linear temporal logic. Experiments in robot navigation and manipulation illustrate that the method is able to generate trajectories that satisfy formulae that specify obstacle avoidance and visitation sequences.
  arXiv  Detail & Related papers  (2024-05-07T11:54:22Z)
• Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference [53.419249906014194]
  We study generative modeling for planning with datasets repurposed from offline reinforcement learning. We introduce the Latent Plan Transformer (), a novel model that leverages a latent variable to connect a Transformer-based trajectory generator and the final return.
  arXiv  Detail & Related papers  (2024-02-07T08:18:09Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Reasoning with Latent Diffusion in Offline Reinforcement Learning [11.349356866928547]
  offline reinforcement learning holds promise as a means to learn high-reward policies from a static dataset. Key challenge in offline RL lies in effectively stitching portions of suboptimal trajectories from the static dataset. We propose a novel approach that leverages the expressiveness of latent diffusion to model in-support trajectory sequences as compressed latent skills.
  arXiv  Detail & Related papers  (2023-09-12T20:58:21Z)
• Learning Minimally-Violating Continuous Control for Infeasible Linear Temporal Logic Specifications [2.496282558123411]
  This paper explores continuous-time control for target-driven navigation to satisfy complex high-level tasks expressed as linear temporal logic (LTL) We propose a model-free synthesis framework using deep reinforcement learning (DRL) where the underlying dynamic system is unknown (an opaque box)
  arXiv  Detail & Related papers  (2022-10-03T18:32:20Z)
• Semi-Supervised Temporal Action Detection with Proposal-Free Masking [134.26292288193298]
  We propose a novel Semi-supervised Temporal action detection model based on PropOsal-free Temporal mask (SPOT) SPOT outperforms state-of-the-art alternatives, often by a large margin.
  arXiv  Detail & Related papers  (2022-07-14T16:58:47Z)
• Model-Based Reinforcement Learning via Latent-Space Collocation [110.04005442935828]
  We argue that it is easier to solve long-horizon tasks by planning sequences of states rather than just actions. We adapt the idea of collocation, which has shown good results on long-horizon tasks in optimal control literature, to the image-based setting by utilizing learned latent state space models.
  arXiv  Detail & Related papers  (2021-06-24T17:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.