Efficient Planning in a Compact Latent Action Space

• URL: http://arxiv.org/abs/2208.10291v2
• Date: Thu, 25 Aug 2022 08:06:32 GMT
• Title: Efficient Planning in a Compact Latent Action Space
• Abstract summary: Trajectory Autoencoding Planner (TAP) is a planning-based sequence modelling RL method that scales to high state-action dimensionalities. Unlike $O(D3)$ complexity of Trajectory Transformer, TAP enjoys constant $O(C)$ planning computational complexity regarding state-action dimensionality.
• Score: 42.40869357513842
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While planning-based sequence modelling methods have shown great potential in continuous control, scaling them to high-dimensional state-action sequences remains an open challenge due to the high computational complexity and innate difficulty of planning in high-dimensional spaces. We propose the Trajectory Autoencoding Planner (TAP), a planning-based sequence modelling RL method that scales to high state-action dimensionalities. Using a state-conditional Vector-Quantized Variational Autoencoder (VQ-VAE), TAP models the conditional distribution of the trajectories given the current state. When deployed as an RL agent, TAP avoids planning step-by-step in a high-dimensional continuous action space but instead looks for the optimal latent code sequences by beam search. Unlike $O(D^3)$ complexity of Trajectory Transformer, TAP enjoys constant $O(C)$ planning computational complexity regarding state-action dimensionality $D$. Our empirical evaluation also shows the increasingly strong performance of TAP with the growing dimensionality. For Adroit robotic hand manipulation tasks with high state and action dimensionality, TAP surpasses existing model-based methods, including TT, with a large margin and also beats strong model-free actor-critic baselines.

Related papers

• Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution [51.519325778407016]
  World models let robots imagine possible futures, but exploiting this capability for real-time control is bottlenecked by a representation misalignment.<n>We present Hydra, a discrete World Action Model that closes this gap by moving the planner, both the sampler and the evaluator, inside the model.<n> Hydra outperforms state-of-the-art world models in goal-directed planning, while matching or exceeding the closed-loop execution capabilities of leading reactive foundation policies.
  arXiv  Detail & Related papers  (2026-08-29T01:58:19Z)
• Dispersive Forward Tree Search for Optimal Control: Coverage, Complexity, and Computation [15.03677748041754]
  This paper develops a propagation-based kinodynamic planner with deterministic finite-sample near-optimality guarantees.<n>We show that a forward tree of locally dispersive control commands contains a near-optimal trajectory at a certified tree size.<n>We implement the resulting search algorithm, Dispersive Forward Tree search (DFT*), as breadth-first expansion of the forward tree.
  arXiv  Detail & Related papers  (2026-08-26T18:46:17Z)
• 3SPO: State-Score-Supervised Policy Optimization for LLM Agents [25.01222941080024]
  We propose textbfState-Score-Supervised Policy Optimization (3SPO), a novel RL algorithm that performs post-step policy optimization with dynamic state score supervision.<n> Experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B-Instruct show that 3SPO consistently outperforms GRPO by $+22.6%$ on ALFWorld and $+15.6$ points on WebShop.
  arXiv  Detail & Related papers  (2026-06-08T14:26:05Z)
• PaT: Planning-after-Trial for Efficient Test-Time Code Generation [25.264318601226265]
  Planning-after-Trial (PaT) is an adaptive policy for code generation that invokes a planner only upon verification failure.<n>We propose PaT, an adaptive policy for code generation that invokes a planner only upon verification failure.
  arXiv  Detail & Related papers  (2026-05-08T05:09:18Z)
• Lifting Embodied World Models for Planning and Control [59.09016913513998]
  We train a lightweight policy that maps high-level actions to sequences of low-level joint actions.<n>We instantiate this framework for a human-like embodiment, defining the high-level action space as a small set of 2D waypoints.<n>We show that the lifted world model substantially outperforms searching directly in low-level joint space.
  arXiv  Detail & Related papers  (2026-04-28T23:59:19Z)
• OOWM: Structuring Embodied Reasoning and Planning via Object-Oriented Programmatic World Modeling [62.3458279176813]
  We propose Object-Oriented World Modeling (OOWM), a novel framework that structures embodied reasoning through the lens of software engineering formalism.<n>OOWM employs Class Diagrams to ground visual perception into rigorous object hierarchies, and Activity Diagrams to operationalize planning into executable control flows.
  arXiv  Detail & Related papers  (2026-02-25T16:01:07Z)
• HiST-VLA: A Hierarchical Spatio-Temporal Vision-Language-Action Model for End-to-End Autonomous Driving [20.266736153749417]
  Vision-Language-Action (VLA) models offer promising capabilities for autonomous driving through multimodal understanding.<n>Their utilization in safety-critical scenarios is constrained by inherent limitations, including numerical reasoning, weak 3D spatial awareness, and high sensitivity to context.<n>We propose HiST-VLA, a novel Hierarchical Spatio-Temporal VLA model designed for reliable trajectory generation.
  arXiv  Detail & Related papers  (2026-02-11T07:08:33Z)
• Tail-Aware Post-Training Quantization for 3D Geometry Models [58.79500829118265]
  Post-Training Quantization (PTQ) enables efficient inference without retraining.<n>PTQ fails to transfer effectively to 3D models due to intricate feature distributions and prohibitive calibration overhead.<n>We propose TAPTQ, a Tail-Aware Post-Training Quantization pipeline for 3D geometric learning.
  arXiv  Detail & Related papers  (2026-02-02T07:21:15Z)
• Parallel Stochastic Gradient-Based Planning for World Models [39.699893143984916]
  We propose a robust and highly parallelizable planner that leverages the differentiability of the learned world model for efficient optimization.<n>Our method treats states as optimization variables ("virtual states") with soft dynamics constraints, enabling parallel and easier optimization.<n>Our planner, which we call GRASP (GradAxed Planner), can be viewed as a valid version of a non-condensed or collocation-based optimal controller.
  arXiv  Detail & Related papers  (2026-01-31T02:57:47Z)
• Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models [68.31088463716269]
  We propose a structured sparse parametrization of transition matrices in state-space models (SSMs)<n>Our method, PD-SSM, parametrizes the transition matrix as the product of a column one-hot matrix ($P$) and a complex-valued diagonal matrix ($D$)<n>The model significantly outperforms a wide collection of modern SSM variants on various FSA state tracking tasks.
  arXiv  Detail & Related papers  (2025-09-26T12:46:30Z)
• Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstraction [7.918703013303246]
  We present Latent Macro Action Planner (L-MAP), which addresses the challenge of learning to make decisions in high-dimensional continuous action spaces. L-MAP learns a set of temporally extended macro-actions through a state-conditional Vector Quantized Variational Autoencoder (VQ-VAE) In offline RL settings, including continuous control tasks, L-MAP efficiently searches over discrete latent actions to yield high expected returns.
  arXiv  Detail & Related papers  (2025-02-28T16:02:23Z)
• Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models [3.5161229331588095]
  Continuous monitoring and real-time control of high-dimensional distributed systems are crucial in applications to ensure a desired physical behavior. Traditional feedback control design that relies on full-order models fails to meet these requirements due to the delay in the control computation. We propose a real-time closed-loop control strategy enhanced by nonlinear non-intrusive Deep Learning-based Reduced Order Models (DL-ROMs)
  arXiv  Detail & Related papers  (2024-12-13T08:04:21Z)
• Towards aerodynamic surrogate modeling based on $β$-variational autoencoders [0.0]
  Surrogate models that combine dimensionality reduction and regression techniques are essential to reduce the need for costly high-fidelity computational fluid dynamics data. We propose a surrogate model based on latent space regression to predict pressure distributions on a transonic wing given the flight conditions: Mach number and angle of attack.
  arXiv  Detail & Related papers  (2024-08-09T09:43:10Z)
• GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling [0.0]
  We develop GateLoop, a sequence model that generalizes linear recurrent models such as S4, S5, LRU and RetNet. GateLoop empirically outperforms existing models for auto-regressive language modeling. We prove that our approach can be interpreted as providing data-controlled relative-positional information to Attention.
  arXiv  Detail & Related papers  (2023-11-03T14:08:39Z)
• COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL [50.385005413810084]
  Dyna-style model-based reinforcement learning contains two phases: model rollouts to generate sample for policy learning and real environment exploration. $textttCOPlanner$ is a planning-driven framework for model-based methods to address the inaccurately learned dynamics model problem.
  arXiv  Detail & Related papers  (2023-10-11T06:10:07Z)
• DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability [58.75803543245372]
  Task and Motion Planning (TAMP) approaches are suited for planning multi-step autonomous robot manipulation. We propose to overcome these limitations by composing diffusion models using a TAMP system. We show how the combination of classical TAMP, generative modeling, and latent embedding enables multi-step constraint-based reasoning.
  arXiv  Detail & Related papers  (2023-06-22T20:40:24Z)
• LocATe: End-to-end Localization of Actions in 3D with Transformers [91.28982770522329]
  LocATe is an end-to-end approach that jointly localizes and recognizes actions in a 3D sequence. Unlike transformer-based object-detection and classification models which consider image or patch features as input, LocATe's transformer model is capable of capturing long-term correlations between actions in a sequence. We introduce a new, challenging, and more realistic benchmark dataset, BABEL-TAL-20 (BT20), where the performance of state-of-the-art methods is significantly worse.
  arXiv  Detail & Related papers  (2022-03-21T03:35:32Z)
• Efficiently Modeling Long Sequences with Structured State Spaces [15.456254157293836]
  We propose a new sequence model based on a new parameterization for the fundamental state space model. S4 achieves strong empirical results across a diverse range of established benchmarks, including (i) 91% accuracy on sequential CIFAR-10 with no data augmentation or auxiliary losses, on par with a larger 2-D ResNet.
  arXiv  Detail & Related papers  (2021-10-31T03:32:18Z)
• Finite-time System Identification and Adaptive Control in Autoregressive Exogenous Systems [79.67879934935661]
  We study the problem of system identification and adaptive control of unknown ARX systems. We provide finite-time learning guarantees for the ARX systems under both open-loop and closed-loop data collection.
  arXiv  Detail & Related papers  (2021-08-26T18:00:00Z)
• Controlled Gaussian Process Dynamical Models with Application to Robotic Cloth Manipulation [10.04778213256535]
  We propose Controlled Gaussian Process Dynamical Model (CGPDM) for learning high-dimensional, nonlinear dynamics. CGPDM is constituted by a low-dimensional latent space, with an associated dynamics where external control variables can act. It is capable of generalizing over a wide range of movements and confidently predicting the cloth motions obtained by previously unseen sequences of control actions.
  arXiv  Detail & Related papers  (2021-03-11T11:34:12Z)
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic [59.94347858883343]
  This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states.
  arXiv  Detail & Related papers  (2021-02-24T01:11:25Z)
• Continuous Motion Planning with Temporal Logic Specifications using Deep Neural Networks [16.296473750342464]
  We propose a model-free reinforcement learning method to synthesize control policies for motion planning problems. The robot is modelled as a discrete Markovtime decision process (MDP) with continuous state and action spaces. We train deep neural networks to approximate the value function and policy using an actorcritic reinforcement learning method.
  arXiv  Detail & Related papers  (2020-04-02T17:58:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.