Potential Field Guided Actor-Critic Reinforcement Learning

• URL: http://arxiv.org/abs/2006.06923v1
• Date: Fri, 12 Jun 2020 03:09:25 GMT
• Title: Potential Field Guided Actor-Critic Reinforcement Learning
• Abstract summary: We combine the reward-based critic with a potential-field-based critic to formulate the proposed potential field guided actor-critic reinforcement learning approach (actor-critic-2) This can be seen as a combination of the model-based gradients and the model-free gradients in policy improvement. Experiments on predator-prey game demonstrate the effectiveness of the proposed approach.
• Score: 0.30458514384586394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we consider the problem of actor-critic reinforcement learning. Firstly, we extend the actor-critic architecture to actor-critic-N architecture by introducing more critics beyond rewards. Secondly, we combine the reward-based critic with a potential-field-based critic to formulate the proposed potential field guided actor-critic reinforcement learning approach (actor-critic-2). This can be seen as a combination of the model-based gradients and the model-free gradients in policy improvement. State with large potential field often contains a strong prior information, such as pointing to the target at a long distance or avoiding collision by the side of an obstacle. In this situation, we should trust potential-field-based critic more as policy evaluation to accelerate policy improvement, where action policy tends to be guided. For example, in practical application, learning to avoid obstacles should be guided rather than learned by trial and error. State with small potential filed is often lack of information, for example, at the local minimum point or around the moving target. At this time, we should trust reward-based critic as policy evaluation more to evaluate the long-term return. In this case, action policy tends to explore. In addition, potential field evaluation can be combined with planning to estimate a better state value function. In this way, reward design can focus more on the final stage of reward, rather than reward shaping or phased reward. Furthermore, potential field evaluation can make up for the lack of communication in multi-agent cooperation problem, i.e., multi-agent each has a reward-based critic and a relative unified potential-field-based critic with prior information. Thirdly, simplified experiments on predator-prey game demonstrate the effectiveness of the proposed approach.

Related papers

• Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generation [73.11410167573614]
  In critique-guided refinement, a critic gives feedback on an initial response and an actor revises it.<n>We frame critique as actor-conditioned revision guidance, where usefulness depends on whether the feedback helps the target actor address the intended weakness.<n>We introduce TAIScore, a reward that evaluates the instruction, initial response, critique, and revision together, assessing whether the critique targets a real weakness, whether the actor follows it, and whether the intended aspect improves.<n>Experiments show that an 8B critic trained with TAIScore outperforms both a zero-shot 120B critic and critics trained with outcome-only or critique-only reward
  arXiv  Detail & Related papers  (2026-08-31T07:50:57Z)
• Decoupling Policy Extraction for Offline Reinforcement Learning [18.247325284604553]
  offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions.<n>We revisit the conventional offline RL paradigm and propose decoupling policy improvement from actor training.<n>Specifically, we train the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic.
  arXiv  Detail & Related papers  (2026-08-21T09:26:53Z)
• VIMPO: Value-Implicit Policy Optimization for LLMs [106.88933849641272]
  Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token.<n>Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability.<n>We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from optimality conditions of KL-regularized reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-18T09:44:12Z)
• Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation [56.84819098277464]
  CoNL is a framework that unifies generation, evaluation, and meta-evaluation through multi-agent self-play.<n>CoNL achieves consistent improvements over self-rewarding baselines while maintaining stable training.
  arXiv  Detail & Related papers  (2026-01-29T09:41:14Z)
• Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning [89.60378227969643]
  We propose Critique-RL, an online RL approach for developing critiquing language models without stronger supervision.<n>Our approach operates on a two-player paradigm: the actor generates a response, the critic provides feedback, and the actor refines the response accordingly.<n>Experiments across various tasks and models show that Critique-RL delivers substantial performance improvements.
  arXiv  Detail & Related papers  (2025-10-28T11:37:01Z)
• Functional Critic Modeling for Provably Convergent Off-Policy Actor-Critic [29.711769434073755]
  We introduce a novel concept of functional critic modeling, which leads to a new AC framework.<n>We provide a theoretical analysis in the linear function setting, establishing the provable convergence of our framework.
  arXiv  Detail & Related papers  (2025-09-26T21:55:26Z)
• Evaluation-Aware Reinforcement Learning [10.594563233900004]
  Policy evaluation is often a prerequisite for deploying safety- and performance-critical systems.<n>We propose evaluation-aware reinforcement learning (EvA-RL), in which a policy is trained to maximize expected return.<n>We show that EvA-RL can substantially reduce evaluation error while maintaining competitive returns.
  arXiv  Detail & Related papers  (2025-09-23T18:17:21Z)
• Optimistic critics can empower small actors [14.058002772699044]
  We argue for the advantages of asymmetric setups, specifically with the use of smaller actors.<n>We find that, in general, smaller actors result in performance degradation and overfit critics.<n>Our analyses suggest poor data collection, due to value underestimation, as one of the main causes for this behavior.
  arXiv  Detail & Related papers  (2025-06-01T14:00:03Z)
• The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement [49.687224320842105]
  Large language models (LLMs) have recently transformed from text-based assistants to autonomous agents capable of planning, reasoning, and iteratively improving their actions. In this work, we introduce Critique-Guided Improvement (CGI), a novel two-player framework, comprising an actor model that explores an environment and a critic model that generates detailed nature language feedback.
  arXiv  Detail & Related papers  (2025-03-20T10:42:33Z)
• LLM-Safety Evaluations Lack Robustness [58.334290876531036]
  We argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise. We propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers.
  arXiv  Detail & Related papers  (2025-03-04T12:55:07Z)
• Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives [52.863024096759816]
  Misaligned research objectives have hindered progress in adversarial robustness research over the past decade. We argue that realigned objectives are necessary for meaningful progress in adversarial alignment.
  arXiv  Detail & Related papers  (2025-02-17T15:28:40Z)
• Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees [12.259191000019033]
  Actor-critic (AC) methods are widely used in reinforcement learning (RL) We design a joint objective for training the actor and critic in a decision-aware fashion. We empirically demonstrate the benefit of our decision-aware actor-critic framework on simple RL problems.
  arXiv  Detail & Related papers  (2023-05-24T15:34:21Z)
• Off-policy evaluation for learning-to-rank via interpolating the item-position model and the position-based model [83.83064559894989]
  A critical need for industrial recommender systems is the ability to evaluate recommendation policies offline, before deploying them to production. We develop a new estimator that mitigates the problems of the two most popular off-policy estimators for rankings. In particular, the new estimator, called INTERPOL, addresses the bias of a potentially misspecified position-based model.
  arXiv  Detail & Related papers  (2022-10-15T17:22:30Z)
• A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning [17.36759906285316]
  We show that state-based critics can introduce bias in the policy estimates, potentially undermining the guarantees of the algorithm. We also show that, even if the state-based critics do not introduce any bias, they can still result in a larger gradient variance, contrary to the common intuition.
  arXiv  Detail & Related papers  (2022-01-03T14:51:30Z)
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction [73.77593805292194]
  We train a separate exploration policy to maximize an approximate upper confidence bound of the critics in an off-policy actor-critic framework. To mitigate the off-policy-ness, we adapt the recently introduced DICE framework to learn a distribution correction ratio for off-policy actor-critic training.
  arXiv  Detail & Related papers  (2021-10-22T22:07:51Z)
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation [53.83642844626703]
  We provide a unifying framework for estimating higher-order derivatives of value functions, based on off-policy evaluation. Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates.
  arXiv  Detail & Related papers  (2021-06-24T15:58:01Z)
• Reliable Off-policy Evaluation for Reinforcement Learning [53.486680020852724]
  In a sequential decision-making problem, off-policy evaluation estimates the expected cumulative reward of a target policy. We propose a novel framework that provides robust and optimistic cumulative reward estimates using one or multiple logged data.
  arXiv  Detail & Related papers  (2020-11-08T23:16:19Z)
• Learning Value Functions in Deep Policy Gradients using Residual Variance [22.414430270991005]
  Policy gradient algorithms have proven to be successful in diverse decision making and control tasks. Traditional actor-critic algorithms do not succeed in fitting the true value function. We provide a new state-value (resp. state-action-value) function approximation that learns the value of the states relative to their mean value.
  arXiv  Detail & Related papers  (2020-10-09T08:57:06Z)
• Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment [61.24399136715106]
  We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. Our focus is on designing methods for detection of such manipulations. Specifically, we consider a setting in which agents evaluate a subset of their peers and output rankings that are later aggregated to form a final ordering.
  arXiv  Detail & Related papers  (2020-10-08T15:08:40Z)
• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization [10.424426548124696]
  We propose MAGE, a model-based actor-critic algorithm, grounded in the theory of policy gradients. MAGE backpropagates through the learned dynamics to compute gradient targets in temporal difference learning. We demonstrate the efficiency of the algorithm in comparison to model-free and model-based state-of-the-art baselines.
  arXiv  Detail & Related papers  (2020-04-29T16:30:53Z)
• Online Meta-Critic Learning for Off-Policy Actor-Critic Methods [107.98781730288897]
  Off-Policy Actor-Critic (Off-PAC) methods have proven successful in a variety of continuous control tasks. We introduce a novel and flexible meta-critic that observes the learning process and meta-learns an additional loss for the actor.
  arXiv  Detail & Related papers  (2020-03-11T14:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.