Marginal Utility for Planning in Continuous or Large Discrete Action Spaces

• URL: http://arxiv.org/abs/2006.06054v2
• Date: Wed, 17 Jun 2020 17:00:39 GMT
• Title: Marginal Utility for Planning in Continuous or Large Discrete Action Spaces
• Abstract summary: Sample-based planning is a powerful family of algorithms for generating intelligent behavior from a model of the environment. Generating good candidate actions is critical to the success of sample-based planners. We show that a generator trained with the marginal utility outperforms hand-coded schemes built on substantial domain knowledge.
• Score: 34.83772836920873
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sample-based planning is a powerful family of algorithms for generating intelligent behavior from a model of the environment. Generating good candidate actions is critical to the success of sample-based planners, particularly in continuous or large action spaces. Typically, candidate action generation exhausts the action space, uses domain knowledge, or more recently, involves learning a stochastic policy to provide such search guidance. In this paper we explore explicitly learning a candidate action generator by optimizing a novel objective, marginal utility. The marginal utility of an action generator measures the increase in value of an action over previously generated actions. We validate our approach in both curling, a challenging stochastic domain with continuous state and action spaces, and a location game with a discrete but large action space. We show that a generator trained with the marginal utility objective outperforms hand-coded schemes built on substantial domain knowledge, trained stochastic policies, and other natural objectives for generating actions for sampled-based planners.

Related papers

• Active Timepoint Selection for Learning Measure-Valued Trajectories [55.49809248697036]
  Inferring continuous probability paths from snapshots is a fundamental challenge in domains like single-cell biology.<n>We introduce a framework which extends active experimentation to the space of sparse measures.<n>This yields an acquisition policy that iteratively selects measurement times to minimize uncertainty.
  arXiv  Detail & Related papers  (2026-05-28T22:22:35Z)
• Using Common Random Numbers for Simulation-based Planning with Rollouts [3.285499181338201]
  Simulation-based planning with rollouts is a widely-deployed technique for decision making in environments.<n>In this paper, we examine the effect of using common random numbers in the simulation process.<n>We obtain a simple recipe for (provably) reducing variance in relative utility when simulations invoke a rollout policy beyond some depth.
  arXiv  Detail & Related papers  (2026-05-06T10:31:18Z)
• Representative Action Selection for Large Action Space: From Bandits to MDPs [47.980675309210746]
  We study the problem of selecting a small, representative action subset from an extremely large action space shared across a family of reinforcement learning (RL) environments.<n>Our goal is to identify a fixed subset of actions that, for every environment in the family, contains a near-optimal action, thereby enabling efficient learning without exhaustively evaluating all actions.
  arXiv  Detail & Related papers  (2025-11-27T04:49:23Z)
• Guided Exploration for Efficient Relational Model Learning [20.853275384388805]
  Efficient exploration is critical for learning relational models in large-scale environments with complex, long-horizon tasks. Goal-literal babbling (GLIB) improves upon random exploration by setting and planning to novel goals. We introduce Baking-Large, a challenging domain with extensive state-action spaces and long-horizon tasks.
  arXiv  Detail & Related papers  (2025-02-10T04:23:01Z)
• Action abstractions for amortized sampling [49.384037138511246]
  We propose an approach to incorporate the discovery of action abstractions, or high-level actions, into the policy optimization process. Our approach involves iteratively extracting action subsequences commonly used across many high-reward trajectories and chunking' them into a single action that is added to the action space.
  arXiv  Detail & Related papers  (2024-10-19T19:22:50Z)
• Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policies [25.760946763103483]
  We propose Imagination Policy, a novel multi-task key-frame policy network for solving high-precision pick and place tasks. Instead of learning actions directly, Imagination Policy generates point clouds to imagine desired states which are then translated to actions using rigid action estimation.
  arXiv  Detail & Related papers  (2024-06-17T17:00:41Z)
• Goal-conditioned Offline Planning from Curious Exploration [28.953718733443143]
  We consider the challenge of extracting goal-conditioned behavior from the products of unsupervised exploration techniques. We find that conventional goal-conditioned reinforcement learning approaches for extracting a value function and policy fall short in this difficult offline setting. In order to mitigate their occurrence, we propose to combine model-based planning over learned value landscapes with a graph-based value aggregation scheme.
  arXiv  Detail & Related papers  (2023-11-28T17:48:18Z)
• AI planning in the imagination: High-level planning on learned abstract search spaces [68.75684174531962]
  We propose a new method, called PiZero, that gives an agent the ability to plan in an abstract search space that the agent learns during training. We evaluate our method on multiple domains, including the traveling salesman problem, Sokoban, 2048, the facility location problem, and Pacman.
  arXiv  Detail & Related papers  (2023-08-16T22:47:16Z)
• Reparameterized Policy Learning for Multimodal Trajectory Optimization [61.13228961771765]
  We investigate the challenge of parametrizing policies for reinforcement learning in high-dimensional continuous action spaces. We propose a principled framework that models the continuous RL policy as a generative model of optimal trajectories. We present a practical model-based RL method, which leverages the multimodal policy parameterization and learned world model.
  arXiv  Detail & Related papers  (2023-07-20T09:05:46Z)
• Exploration via Planning for Information about the Optimal Trajectory [67.33886176127578]
  We develop a method that allows us to plan for exploration while taking the task and the current knowledge into account. We demonstrate that our method learns strong policies with 2x fewer samples than strong exploration baselines.
  arXiv  Detail & Related papers  (2022-10-06T20:28:55Z)
• Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning [29.811723497181486]
  Generative Planning method (GPM) can generate actions not only for the current step, but also for a number of future steps. GPM can leverage its generated multi-step plans for temporally coordinated exploration towards high value regions.
  arXiv  Detail & Related papers  (2022-01-24T15:53:32Z)
• Discrete Action On-Policy Learning with Action-Value Critic [72.20609919995086]
  Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension. We construct a critic to estimate action-value functions, apply it on correlated actions, and combine these critic estimated action values to control the variance of gradient estimation. These efforts result in a new discrete action on-policy RL algorithm that empirically outperforms related on-policy algorithms relying on variance control techniques.
  arXiv  Detail & Related papers  (2020-02-10T04:23:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.