Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning

• URL: http://arxiv.org/abs/2201.09765v1
• Date: Mon, 24 Jan 2022 15:53:32 GMT
• Title: Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning
• Abstract summary: Generative Planning method (GPM) can generate actions not only for the current step, but also for a number of future steps. GPM can leverage its generated multi-step plans for temporally coordinated exploration towards high value regions.
• Score: 29.811723497181486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard model-free reinforcement learning algorithms optimize a policy that generates the action to be taken in the current time step in order to maximize expected future return. While flexible, it faces difficulties arising from the inefficient exploration due to its single step nature. In this work, we present Generative Planning method (GPM), which can generate actions not only for the current step, but also for a number of future steps (thus termed as generative planning). This brings several benefits to GPM. Firstly, since GPM is trained by maximizing value, the plans generated from it can be regarded as intentional action sequences for reaching high value regions. GPM can therefore leverage its generated multi-step plans for temporally coordinated exploration towards high value regions, which is potentially more effective than a sequence of actions generated by perturbing each action at single step level, whose consistent movement decays exponentially with the number of exploration steps. Secondly, starting from a crude initial plan generator, GPM can refine it to be adaptive to the task, which, in return, benefits future explorations. This is potentially more effective than commonly used action-repeat strategy, which is non-adaptive in its form of plans. Additionally, since the multi-step plan can be interpreted as the intent of the agent from now to a span of time period into the future, it offers a more informative and intuitive signal for interpretation. Experiments are conducted on several benchmark environments and the results demonstrated its effectiveness compared with several baseline methods.

Related papers

• Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models [101.44987067974999]
  Coarse-to-Control is a plan-execute VLA that introduces planning in the action-token space.<n>We show that action-token planning consistently improves over direct action generation.
  arXiv  Detail & Related papers  (2026-06-05T10:01:37Z)
• Spatially Grounded Long-Horizon Task Planning in the Wild [49.614308474374035]
  GroundedPlanBench is a newly curated benchmark for spatially grounded long-horizon action planning in the wild.<n>Video-to-Spatially Grounded Planning (V2GP) is an automated data generation framework that leverages real-world robot video demonstrations to improve spatially grounded long-horizon planning.
  arXiv  Detail & Related papers  (2026-03-13T05:42:18Z)
• Robot Local Planner: A Periodic Sampling-Based Motion Planner with Minimal Waypoints for Home Environments [0.0]
  We propose a periodic sampling-based whole-body trajectory planning method, called the "Robot Local Planner"<n>The RLP minimizes computation time by planning with minimal waypoints and generating safe trajectories.<n> Evaluation experiments demonstrated that the RLP outperformed existing methods in terms of motion planning time, motion duration, and robustness.
  arXiv  Detail & Related papers  (2026-02-24T07:46:31Z)
• The Cognitive Bandwidth Bottleneck: Shifting Long-Horizon Agent from Planning with Actions to Planning with Schemas [56.62286434195321]
  This paper systematically studies the effectiveness of two different action representations.<n>We propose cognitive bandwidth perspective as a conceptual framework to qualitatively understand the differences.<n>We provide an actionable guide for building more capable PwS agents for better scalable autonomy.
  arXiv  Detail & Related papers  (2025-10-08T14:47:40Z)
• Improving planning and MBRL with temporally-extended actions [2.7777488754991206]
  Continuous time systems are often modeled using discrete time dynamics but this requires a small simulation step to maintain accuracy.<n>Previous work in model free reinforcement learning has partially addressed this issue using action repeats where a policy is learned to determine a discrete action duration.<n>We propose to control the continuous decision timescale directly by using temporally-extended actions and letting the planner treat the duration of the action as an additional optimization variable.
  arXiv  Detail & Related papers  (2025-05-21T16:59:32Z)
• Efficient Robotic Policy Learning via Latent Space Backward Planning [17.770562202624962]
  Current robotic planning methods often rely on predicting multi-frame images with full pixel details.<n>We propose a Latent Space Backward Planning scheme (LBP), which begins by grounding the task into final latent goals.<n>We show that LBP outperforms existing fine-grained and forward planning methods, achieving SOTA performance.
  arXiv  Detail & Related papers  (2025-05-11T06:13:51Z)
• Variable Time-Step MPC for Agile Multi-Rotor UAV Interception of Dynamic Targets [6.0967385124149756]
  Agile planning using existing non-linear model predictive control methods is limited by the number of planning steps as it becomes increasingly demanding. In this paper, we propose to address these limitations by introducing variable time steps and coupling them with the prediction horizon length. A simplified point-mass motion primitive is used to leverage the differential flatness of quadrotor dynamics and the trajectory generation of feasible trajectories in the flat output space.
  arXiv  Detail & Related papers  (2025-03-18T11:59:24Z)
• Adaptive Planning with Generative Models under Uncertainty [20.922248169620783]
  Planning with generative models has emerged as an effective decision-making paradigm across a wide range of domains. While continuous replanning at each timestep might seem intuitive because it allows decisions to be made based on the most recent environmental observations, it results in substantial computational challenges. Our work addresses this challenge by introducing a simple adaptive planning policy that leverages the generative model's ability to predict long-horizon state trajectories.
  arXiv  Detail & Related papers  (2024-08-02T18:07:53Z)
• AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation [89.68433168477227]
  Large Language Model (LLM) based agents have garnered significant attention and are becoming increasingly popular. This paper investigates enhancing the planning abilities of LLMs through instruction tuning. To address this limitation, this paper explores the automated synthesis of diverse environments and a gradual range of planning tasks.
  arXiv  Detail & Related papers  (2024-08-01T17:59:46Z)
• AI planning in the imagination: High-level planning on learned abstract search spaces [68.75684174531962]
  We propose a new method, called PiZero, that gives an agent the ability to plan in an abstract search space that the agent learns during training. We evaluate our method on multiple domains, including the traveling salesman problem, Sokoban, 2048, the facility location problem, and Pacman.
  arXiv  Detail & Related papers  (2023-08-16T22:47:16Z)
• Planning with Sequence Models through Iterative Energy Minimization [22.594413287842574]
  We suggest an approach towards integrating planning with sequence models based on the idea of iterative energy minimization. We train a masked language model to capture an implicit energy function over trajectories of actions, and formulate planning as finding a trajectory of actions with minimum energy. We illustrate how this procedure enables improved performance over recent approaches across BabyAI and Atari environments.
  arXiv  Detail & Related papers  (2023-03-28T17:53:22Z)
• Achieving mouse-level strategic evasion performance using real-time computational planning [59.60094442546867]
  Planning is an extraordinary ability in which the brain imagines and then enacts evaluated possible futures. We develop a more efficient biologically-inspired planning algorithm, TLPPO, based on work on how the ecology of an animal governs the value of spatial planning. We compare the performance of a real-time agent using TLPPO against the performance of live mice, all tasked with evading a robot predator.
  arXiv  Detail & Related papers  (2022-11-04T18:34:36Z)
• Active Learning of Abstract Plan Feasibility [17.689758291966502]
  We present an active learning approach to efficiently acquire an APF predictor through task-independent, curious exploration on a robot. We leverage an infeasible subsequence property to prune candidate plans in the active learning strategy, allowing our system to learn from less data. In a stacking domain where objects have non-uniform mass distributions, we show that our system permits real robot learning of an APF model in four hundred self-supervised interactions.
  arXiv  Detail & Related papers  (2021-07-01T18:17:01Z)
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors [124.30562402952319]
  The ability to predict and plan into the future is fundamental for agents acting in the world. Current learning approaches for visual prediction and planning fail on long-horizon tasks. We propose a framework for visual prediction and planning that is able to overcome both of these limitations.
  arXiv  Detail & Related papers  (2020-06-23T17:58:56Z)
• Zeroth-Order Supervised Policy Improvement [94.0748002906652]
  Policy gradient (PG) algorithms have been widely used in reinforcement learning (RL) We propose Zeroth-Order Supervised Policy Improvement (ZOSPI) ZOSPI exploits the estimated value function $Q$ globally while preserving the local exploitation of the PG methods.
  arXiv  Detail & Related papers  (2020-06-11T16:49:23Z)
• Marginal Utility for Planning in Continuous or Large Discrete Action Spaces [34.83772836920873]
  Sample-based planning is a powerful family of algorithms for generating intelligent behavior from a model of the environment. Generating good candidate actions is critical to the success of sample-based planners. We show that a generator trained with the marginal utility outperforms hand-coded schemes built on substantial domain knowledge.
  arXiv  Detail & Related papers  (2020-06-10T20:24:53Z)
• The Importance of Prior Knowledge in Precise Multimodal Prediction [71.74884391209955]
  Roads have well defined geometries, topologies, and traffic rules. In this paper we propose to incorporate structured priors as a loss function. We demonstrate the effectiveness of our approach on real-world self-driving datasets.
  arXiv  Detail & Related papers  (2020-06-04T03:56:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.