Co-Evolving LLM Evaluators and Policies via DynamicRubric

• URL: http://arxiv.org/abs/2607.20083v2
• Date: Thu, 23 Jul 2026 06:34:29 GMT
• Title: Co-Evolving LLM Evaluators and Policies via DynamicRubric
• Abstract summary: Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models.<n>We propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework.<n>In experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines.
• Score: 67.2962847914162
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models. As policies improve, these sampled responses become close in quality. These close candidates create a bottleneck for policy optimization: collapsed relative evaluator score gaps yield weak or misleading policy supervision. We theoretically characterize why these gaps matter through a probability allocation view, showing that the directional gain of shifting probability mass from one response to another is exactly the evaluator score gap between them. This identifies relative score gaps as the policy optimization signals that guide updates. Motivated by this view, we propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework that generates weighted binary rubric items for each candidate set and aggregates the resulting judgments into response-level scores. In our experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines using a 70B reward model or a 235B static rubric generator. DynamicRubric-optimized policies also show gains on verifiable reasoning and coding tasks. A DynamicRubric-optimized model is fully deployed in WeChat Search's AI answering scenario, where it serves all online traffic across tens of millions of requests per day and improves key online metrics. These results suggest a principle for evaluator-guided post-training: evaluators should evolve with the policies they supervise.

Related papers

• SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning [14.505575259468053]
  Test-time reinforcement learning (TTRL) enables language models to self-evolve at inference time without labeled feedback.<n>We introduce SERPO, which replaces answer voting with a closed loop that co-evolves response evidence, querytokens, and policy parameters.<n>New actor rollouts then refresh both the archives and rubrics, closing the three-way evolution loop.
  arXiv  Detail & Related papers  (2026-07-29T13:03:07Z)
• SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning [38.54413500261524]
  Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions.<n>We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills.
  arXiv  Detail & Related papers  (2026-07-16T09:57:18Z)
• EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments [64.33452656100393]
  We introduce Autonomous Policy Evolution, a controlled evaluation setting in which a harness-model agent repeatedly edits an executable policy system under a fixed interaction budget.<n>We instantiate this setting in EvoPolicyGym, a benchmark built from compact interactive RL environments.<n>On the EvoPolicyGym suite, GPT-5.5 achieves the strongest aggregate rank score and top-two performance on all 16 environments.
  arXiv  Detail & Related papers  (2026-07-02T17:10:13Z)
• EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning [22.209572966965165]
  EvoRubrics is a co-evolutionary framework where a Policy Generator and a RL Generator jointly improve through adversarial interaction within each training step.<n>EvoRubrics consistently outperforms static and dynamic rubrics across benchmarks.
  arXiv  Detail & Related papers  (2026-06-22T08:46:23Z)
• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• Self-evolving LLM agents with in-distribution Optimization [60.05867547965365]
  Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments.<n>We propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning.<n>We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance.
  arXiv  Detail & Related papers  (2026-06-05T15:09:52Z)
• The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement [32.29319213544292]
  Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data.<n>We propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training.
  arXiv  Detail & Related papers  (2026-05-29T06:25:16Z)
• Off-Policy Evaluation for Ranking Policies under Deterministic Logging Policies [17.672063433232196]
  Off-Policy Evaluation (OPE) is an important practical problem in ranking systems.<n>The goal is to estimate the expected performance of a new ranking policy using only logged offline data collected under a different, logging policy.<n>We propose novel estimators, Click-based Inverse Propensity Score (CIPS), exploiting the intrinsicity of user click behavior to address this challenge.
  arXiv  Detail & Related papers  (2026-03-23T02:13:39Z)
• Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning [52.97053840476386]
  We show that well-designed behaviour policies can be used to collect off-policy data for provably lower variance return estimates.<n>We extend this key insight to the online reinforcement learning setting, where both policy evaluation and improvement are interleaved.
  arXiv  Detail & Related papers  (2025-11-13T23:06:40Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.