Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generation

• URL: http://arxiv.org/abs/2608.30397v1
• Date: Mon, 31 Aug 2026 07:50:57 GMT
• Title: Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generation
• Abstract summary: In critique-guided refinement, a critic gives feedback on an initial response and an actor revises it.<n>We frame critique as actor-conditioned revision guidance, where usefulness depends on whether the feedback helps the target actor address the intended weakness.<n>We introduce TAIScore, a reward that evaluates the instruction, initial response, critique, and revision together, assessing whether the critique targets a real weakness, whether the actor follows it, and whether the intended aspect improves.<n>Experiments show that an 8B critic trained with TAIScore outperforms both a zero-shot 120B critic and critics trained with outcome-only or critique-only reward
• Score: 73.11410167573614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Natural-language critiques provide supervision beyond scalar rewards for non-verifiable generation, which lacks deterministic verifiers. In critique-guided refinement, a critic gives feedback on an initial response and an actor revises it. However, final revision quality does not reveal whether the critique was actually useful: a capable actor may improve without following the feedback, while valid feedback may fail if the actor cannot execute it. We frame critique as actor-conditioned revision guidance, where usefulness depends on whether the feedback helps the target actor address the intended weakness. We introduce TAIScore (Targeted Actionable Improvement Score), a reward that evaluates the instruction, initial response, critique, and revision together, assessing whether the critique targets a real weakness, whether the actor follows it, and whether the intended aspect improves. We use this reward to train an actor-tailored critic with GRPO, and use critique-guided refinements to construct DPO preference pairs for the actor, forming a co-evolving critic-actor loop where the critic adapts to the actor's changing capability. Experiments show that an 8B critic trained with TAIScore outperforms both a zero-shot 120B critic and critics trained with outcome-only or critique-only reward signals. Co-evolving the critic and actor further improves performance, suggesting that effective critique supervision should adapt as the actor changes.

Related papers

• Refined Analysis of Entropy-Regularized Actor-Critic [26.419221337454008]
  We study the role of the critic in actor-critic for entropy-regularized, finite discounted environments.<n>This suggests to learn the critic first, keeping it up to date after each actor update.
  arXiv  Detail & Related papers  (2026-05-23T02:41:07Z)
• Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation [56.84819098277464]
  CoNL is a framework that unifies generation, evaluation, and meta-evaluation through multi-agent self-play.<n>CoNL achieves consistent improvements over self-rewarding baselines while maintaining stable training.
  arXiv  Detail & Related papers  (2026-01-29T09:41:14Z)
• Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning [89.60378227969643]
  We propose Critique-RL, an online RL approach for developing critiquing language models without stronger supervision.<n>Our approach operates on a two-player paradigm: the actor generates a response, the critic provides feedback, and the actor refines the response accordingly.<n>Experiments across various tasks and models show that Critique-RL delivers substantial performance improvements.
  arXiv  Detail & Related papers  (2025-10-28T11:37:01Z)
• Training Language Model to Critique for Better Refinement [58.73039433159486]
  We introduce textbfRefinement-oriented textbfCritique textbfOptimization (RCO), a novel framework designed to train critic models using refinement signals.<n>RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses.<n>By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment.
  arXiv  Detail & Related papers  (2025-06-27T12:10:57Z)
• The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement [61.00950725408354]
  Large language models (LLMs) have recently transformed from text-based assistants to autonomous agents capable of planning, reasoning, and iteratively improving their actions.<n>In this work, we introduce Critique-Guided Improvement (CGI), a novel two-player framework, comprising an actor model that explores an environment and a critic model that generates detailed nature language feedback.
  arXiv  Detail & Related papers  (2025-03-20T10:42:33Z)
• Self-Evolving Critique Abilities in Large Language Models [59.861013614500024]
  This paper explores enhancing critique abilities of Large Language Models (LLMs)<n>We introduce SCRIT, a framework that trains LLMs with self-generated data to evolve their critique abilities.<n>Our analysis reveals that SCRIT's performance scales positively with data and model size.
  arXiv  Detail & Related papers  (2025-01-10T05:51:52Z)
• VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning [112.35483894933904]
  We propose VISCO, the first benchmark to extensively analyze the fine-grained critique and correction capabilities of LVLMs.<n>VISCO features dense and fine-grained critique, requiring LVLMs to evaluate the correctness of each step in the chain-of-thought.<n>LookBack significantly improves critique and correction performance by up to 13.5%.
  arXiv  Detail & Related papers  (2024-12-03T05:04:49Z)
• Potential Field Guided Actor-Critic Reinforcement Learning [0.30458514384586394]
  We combine the reward-based critic with a potential-field-based critic to formulate the proposed potential field guided actor-critic reinforcement learning approach (actor-critic-2) This can be seen as a combination of the model-based gradients and the model-free gradients in policy improvement. Experiments on predator-prey game demonstrate the effectiveness of the proposed approach.
  arXiv  Detail & Related papers  (2020-06-12T03:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.