Relating Reinforcement Learning to Dynamic Programming-Based Planning

• URL: http://arxiv.org/abs/2603.07844v1
• Date: Sun, 08 Mar 2026 23:28:23 GMT
• Title: Relating Reinforcement Learning to Dynamic Programming-Based Planning
• Abstract summary: This paper bridges the gap between optimal planning and reinforcement learning (RL)<n>A derandomized version of RL is developed, analyzed, and implemented to yield performance comparisons with value and Dijkstra's algorithm.
• Score: 1.814099487268094
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper bridges some of the gap between optimal planning and reinforcement learning (RL), both of which share roots in dynamic programming applied to sequential decision making or optimal control. Whereas planning typically favors deterministic models, goal termination, and cost minimization, RL tends to favor stochastic models, infinite-horizon discounting, and reward maximization in addition to learning-related parameters such as the learning rate and greediness factor. A derandomized version of RL is developed, analyzed, and implemented to yield performance comparisons with value iteration and Dijkstra's algorithm using simple planning models. Next, mathematical analysis shows: 1) conditions under which cost minimization and reward maximization are equivalent, 2) conditions for equivalence of single-shot goal termination and infinite-horizon episodic learning, and 3) conditions under which discounting causes goal achievement to fail. The paper then advocates for defining and optimizing truecost, rather than inserting arbitrary parameters to guide operations. Performance studies are then extended to the stochastic case, using planning-oriented criteria and comparing value iteration to RL with learning rates and greediness factors.

Related papers

• Decision-Driven Regularization: A Blended Model for Learning and Optimization [1.4055982190455338]
  We study the learning and optimization approach, which first learns how outcomes result from the features, and then selects optimal decisions based on these outcomes.<n>We propose a bi-objective formulation that balances prediction accuracy and cost minimization, termed decision-driven regularization.
  arXiv  Detail & Related papers  (2026-08-15T08:49:33Z)
• PaT: Planning-after-Trial for Efficient Test-Time Code Generation [25.264318601226265]
  Planning-after-Trial (PaT) is an adaptive policy for code generation that invokes a planner only upon verification failure.<n>We propose PaT, an adaptive policy for code generation that invokes a planner only upon verification failure.
  arXiv  Detail & Related papers  (2026-05-08T05:09:18Z)
• Generative Actor Critic [74.04971271003869]
  Generative Actor Critic (GAC) is a novel framework that decouples sequential decision-making by reframing textitpolicy evaluation as learning a generative model of the joint distribution over trajectories and returns.<n>Experiments on Gym-MuJoCo and Maze2D benchmarks demonstrate GAC's strong offline performance and significantly enhanced offline-to-online improvement compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-12-25T06:31:11Z)
• Adaptive Test-Time Reasoning via Reward-Guided Dual-Phase Search [62.1546099504045]
  We propose a dual-phase test-time scaling framework that separates reasoning into planning and execution.<n>Specifically, we decompose reasoning trajectories and develop reward models for each phase, enabling the search to explore and prune plans and executions separately.<n> Experiments on both mathematical reasoning and code generation benchmarks demonstrate that our approach consistently improves accuracy while reducing computation redundant.
  arXiv  Detail & Related papers  (2025-09-29T19:27:23Z)
• Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models [79.2162092822111]
  We systematically evaluate reinforcement learning (RL) and control-based methods on a suite of navigation tasks.<n>We employ a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning.<n>Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts.
  arXiv  Detail & Related papers  (2025-02-20T18:39:41Z)
• Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge [78.28188747489769]
  We propose EvalPlanner, a preference optimization algorithm for Thinking-LLM-as-a-Judge.<n>In a self-training loop, EvalPlanner iteratively optimize over synthetically constructed evaluation plans and executions.<n>Our method achieves a new state-of-the-art performance for generative reward models on RewardBench.
  arXiv  Detail & Related papers  (2025-01-30T02:21:59Z)
• Towards General-Purpose Model-Free Reinforcement Learning [40.973429772093155]
  Reinforcement learning (RL) promises a framework for near-universal problem-solving.<n>In practice, RL algorithms are often tailored to specific benchmarks.<n>We propose a unifying model-free deep RL algorithm that can address a diverse class of domains and problem settings.
  arXiv  Detail & Related papers  (2025-01-27T15:36:37Z)
• Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems [10.404992912881601]
  We study reinforcement learning (RL) for a class of continuous-time linear-quadratic (LQ) control problems for diffusions.<n>We apply a model-free approach that relies neither on knowledge of model parameters nor on their estimations, and devise an RL algorithm to learn the optimal policy parameter directly.
  arXiv  Detail & Related papers  (2024-07-24T12:26:21Z)
• Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration [87.53543137162488]
  We propose an easy-to-implement online reinforcement learning (online RL) framework called textttMEX. textttMEX integrates estimation and planning components while balancing exploration exploitation automatically. It can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards.
  arXiv  Detail & Related papers  (2023-05-29T17:25:26Z)
• Assessment of Reinforcement Learning Algorithms for Nuclear Power Plant Fuel Optimization [0.0]
  This work presents a first-of-a-kind approach to utilize deep RL to solve the loading pattern problem and could be leveraged for any engineering design optimization.
  arXiv  Detail & Related papers  (2023-05-09T23:51:24Z)
• Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation [107.54516740713969]
  We study human-in-the-loop reinforcement learning (RL) with trajectory preferences. Instead of receiving a numeric reward at each step, the agent only receives preferences over trajectory pairs from a human overseer. We propose the first optimistic model-based algorithm for PbRL with general function approximation.
  arXiv  Detail & Related papers  (2022-05-23T09:03:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.