Refined Analysis of Entropy-Regularized Actor-Critic

• URL: http://arxiv.org/abs/2605.24357v1
• Date: Sat, 23 May 2026 02:41:07 GMT
• Title: Refined Analysis of Entropy-Regularized Actor-Critic
• Abstract summary: We study the role of the critic in actor-critic for entropy-regularized, finite discounted environments.<n>This suggests to learn the critic first, keeping it up to date after each actor update.
• Score: 26.419221337454008
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we study the role of the critic in actor--critic for entropy-regularized, finite, discounted environments. We establish that, when the critic is exact, using the latter as a baseline is a variance-reduction method in a strong sense. In this case, actor--critic with stochastic gradients matches the sample complexity of deterministic policy gradient, reaching an $ε$-optimal regularized value with $\tilde{O}(\log(1/ε))$ samples. In practice, the critic is learned alongside the actor: the variance of the actor update is then influenced by the critic's variance and bias. Specifically, when the critic has a sufficiently small error, the variance reduction and rapid convergence are preserved. This suggests to learn the critic first, keeping it up to date after each actor update, underscoring the crucial role of accurate critic estimation in actor--critic methods.

Related papers

• Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generation [73.11410167573614]
  In critique-guided refinement, a critic gives feedback on an initial response and an actor revises it.<n>We frame critique as actor-conditioned revision guidance, where usefulness depends on whether the feedback helps the target actor address the intended weakness.<n>We introduce TAIScore, a reward that evaluates the instruction, initial response, critique, and revision together, assessing whether the critique targets a real weakness, whether the actor follows it, and whether the intended aspect improves.<n>Experiments show that an 8B critic trained with TAIScore outperforms both a zero-shot 120B critic and critics trained with outcome-only or critique-only reward
  arXiv  Detail & Related papers  (2026-08-31T07:50:57Z)
• Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning [0.0]
  Critic complexity is introduced as an additional diagnostic and intervention dimension for actor-critic reinforcement learning.<n>Across TD3 and PPO experiments, critic complexity is tracked together with return and Monte Carlo value-estimation bias.<n>A direct complexity-control intervention is then evaluated by adding a spectral-entropy penalty to the critic loss.
  arXiv  Detail & Related papers  (2026-07-01T05:10:14Z)
• EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training [69.32453275232662]
  We show that a learned critic can inject estimation noise that exceeds the state signal it captures, increasing rather than reducing advantage variance.<n>We propose Explained Variance Policy Optimization (EVPO), which monitors batch-level EV at each training step and adaptively switches between critic-based and batch-mean advantage estimation.
  arXiv  Detail & Related papers  (2026-04-21T14:07:39Z)
• Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty [0.0]
  Off-policy actor-critic methods in reinforcement learning train a critic with temporal-difference updates and use it as a learning signal for the policy (actor)<n>Current methods employ ensembling to quantify the critic's uncertainty-uncertainty due to limited data and model ambiguity-to scale pessimistic updates.<n>In this work, we propose a new algorithm called Actor-C (STAC) that incorporates temporal (one) aleatoric uncertainty-uncertainty arising from transitions, rewards, and policy-induced variability in Bellman.
  arXiv  Detail & Related papers  (2026-01-02T16:33:17Z)
• Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees [12.259191000019033]
  Actor-critic (AC) methods are widely used in reinforcement learning (RL) We design a joint objective for training the actor and critic in a decision-aware fashion. We empirically demonstrate the benefit of our decision-aware actor-critic framework on simple RL problems.
  arXiv  Detail & Related papers  (2023-05-24T15:34:21Z)
• Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Function [69.59204851882643]
  We identify a bias in the meta-gradient of current meta-gradient RL approaches. This bias comes from using the critic that is trained using the meta-learned discount factor for the advantage estimation in the outer objective. Because the meta-learned discount factor is typically lower than the one used in the outer objective, the resulting bias can cause the meta-gradient to favor myopic policies.
  arXiv  Detail & Related papers  (2022-11-19T00:59:20Z)
• Finite-time analysis of single-timescale actor-critic [8.994243376183658]
  Actor-critic methods have achieved significant success in many challenging applications. finite-time convergence is still poorly understood in the most practical single-timescale form. We investigate the more practical online single-timescale actor-critic algorithm on continuous state space.
  arXiv  Detail & Related papers  (2022-10-18T15:03:56Z)
• A Small Gain Analysis of Single Timescale Actor Critic [16.092248433189816]
  We consider a version of actor-critic which uses proportional step-sizes and only one critic update with a single sample from the stationary distribution per actor step. We prove that this method can be used to find a stationary point, and that the resulting sample complexity improves the state of the art for actor-critic methods.
  arXiv  Detail & Related papers  (2022-03-04T22:20:34Z)
• A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms [81.01917016753644]
  We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective. Theoretically, actor-critic algorithms usually have discounting for both actor and critic, i.e., there is a $gammat$ term in the actor update for the transition observed at time $t$ in a trajectory. Practitioners, however, usually ignore the discounting ($gammat$) for the actor while using a discounted critic.
  arXiv  Detail & Related papers  (2020-10-02T15:51:48Z)
• Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy [122.01837436087516]
  We study the global convergence and global optimality of actor-critic, one of the most popular families of reinforcement learning algorithms. We establish the rate of convergence and global optimality of single-timescale actor-critic with linear function approximation for the first time.
  arXiv  Detail & Related papers  (2020-08-02T14:01:49Z)
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics [65.51757376525798]
  Overestimation bias is one of the major impediments to accurate off-policy learning. This paper investigates a novel way to alleviate the overestimation bias in a continuous control setting. Our method---Truncated Quantile Critics, TQC,---blends three ideas: distributional representation of a critic, truncation of critics prediction, and ensembling of multiple critics.
  arXiv  Detail & Related papers  (2020-05-08T19:52:26Z)
• A Finite Time Analysis of Two Time-Scale Actor Critic Methods [87.69128666220016]
  We provide a non-asymptotic analysis for two time-scale actor-critic methods under non-i.i.d. setting. We prove that the actor-critic method is guaranteed to find a first-order stationary point. This is the first work providing finite-time analysis and sample complexity bound for two time-scale actor-critic methods.
  arXiv  Detail & Related papers  (2020-05-04T09:45:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.