Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning

• URL: http://arxiv.org/abs/2607.00452v1
• Date: Wed, 01 Jul 2026 05:10:14 GMT
• Title: Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning
• Abstract summary: Critic complexity is introduced as an additional diagnostic and intervention dimension for actor-critic reinforcement learning.<n>Across TD3 and PPO experiments, critic complexity is tracked together with return and Monte Carlo value-estimation bias.<n>A direct complexity-control intervention is then evaluated by adding a spectral-entropy penalty to the critic loss.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Actor-critic methods depend on learned critics, but critic quality is often evaluated only indirectly through return, temporal-difference error, or value loss. Critic complexity is introduced as an additional diagnostic and intervention dimension for actor-critic reinforcement learning. The analysis uses spectral effective-rank entropy, a rank-like summary of the singular-value distributions of critic weight matrices, to assess critic model complexity. Across TD3 and PPO experiments, critic complexity is tracked together with return and Monte Carlo value-estimation bias. The results show that critic complexity is measurable throughout training and is systematically associated with training behavior, while also making clear that the relationship is heterogeneous across algorithms, tasks, and hyperparameters. A direct complexity-control intervention is then evaluated by adding a spectral-entropy penalty to the critic loss. This intervention reliably changes the targeted spectral quantity, demonstrating that critic complexity can be controlled rather than only observed. Return effects are treated as task-dependent evidence rather than as a general performance claim, because overall complexity-control results vary.

Related papers

• Generalization and Trade-off in Adversarial Training: An RKHS Perspective via Kernel Integral Operators [16.801544027052138]
  Adrial training has emerged as a powerful approach for protecting models against adversarial attacks.<n>We study adversarial training in the robustness reproducing kernel Hilbert space (RKHS) framework through the associated kernel integral operator.<n>We propose a two-stage noise-debiased procedure that estimates and removes the noise contribution from the mixed robustness term.
  arXiv  Detail & Related papers  (2026-07-30T10:43:29Z)
• Refined Analysis of Entropy-Regularized Actor-Critic [26.419221337454008]
  We study the role of the critic in actor-critic for entropy-regularized, finite discounted environments.<n>This suggests to learn the critic first, keeping it up to date after each actor update.
  arXiv  Detail & Related papers  (2026-05-23T02:41:07Z)
• What Makes Value Learning Efficient in Residual Reinforcement Learning? [57.635661297706065]
  Residual reinforcement learning (RL) enables stable online refinement of expressive pretrained policies by freezing the base and learning only bounded corrections.<n>In this work, we identify two key bottlenecks: cold start pathology, where the critic lacks knowledge of the value landscape around the base policy, and structural scale mismatch.<n>We propose DAWN, a minimal approach targeting efficient value learning in residual RL.
  arXiv  Detail & Related papers  (2026-02-11T05:25:39Z)
• TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents [51.30998248590416]
  Trajectory-Aware Comprehensive Evaluation (TRACE) is a framework that holistically assesses the entire problem-solving trajectory.<n>Our contributions include the TRACE framework, its novel metrics, and the accompanying DeepResearch-Bench with controllable complexity.
  arXiv  Detail & Related papers  (2026-02-05T13:28:57Z)
• Bridging Internal Probability and Self-Consistency for Effective and Efficient LLM Reasoning [53.25336975467293]
  We present the first theoretical error decomposition analysis of methods such as perplexity and self-consistency.<n>Our analysis reveals a fundamental trade-off: perplexity methods suffer from substantial model error due to the absence of a proper consistency function.<n>We propose Reasoning-Pruning Perplexity Consistency (RPC), which integrates perplexity with self-consistency, and Reasoning Pruning, which eliminates low-probability reasoning paths.
  arXiv  Detail & Related papers  (2025-02-01T18:09:49Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Effects of Spectral Normalization in Multi-agent Reinforcement Learning [7.064383217512461]
  We show that regularising the critic with spectral normalization (SN) enables it to learn more robustly. Our experiments show that the regularised critic is quickly able to learn from the sparse rewarding experience.
  arXiv  Detail & Related papers  (2022-12-10T16:26:30Z)
• Solving Continuous Control via Q-learning [54.05120662838286]
  We show that a simple modification of deep Q-learning largely alleviates issues with actor-critic methods. By combining bang-bang action discretization with value decomposition, framing single-agent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods.
  arXiv  Detail & Related papers  (2022-10-22T22:55:50Z)
• Stateful Offline Contextual Policy Evaluation and Learning [88.9134799076718]
  We study off-policy evaluation and learning from sequential data. We formalize the relevant causal structure of problems such as dynamic personalized pricing. We show improved out-of-sample policy performance in this class of relevant problems.
  arXiv  Detail & Related papers  (2021-10-19T16:15:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.