EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

• URL: http://arxiv.org/abs/2606.23038v1
• Date: Mon, 22 Jun 2026 08:46:23 GMT
• Title: EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
• Abstract summary: EvoRubrics is a co-evolutionary framework where a Policy Generator and a RL Generator jointly improve through adversarial interaction within each training step.<n>EvoRubrics consistently outperforms static and dynamic rubrics across benchmarks.
• Score: 22.209572966965165
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rubric-based rewards offer interpretable and fine-grained optimization signals for reinforcement learning in open-ended tasks where verifiable answers are unavailable. However, pre-constructed rubrics remain static throughout training, creating a fundamental mismatch with the evolving policy: fixed criteria gradually lose discriminative power as the model improves, leading to reward saturation and potential hacking. Recent dynamic rubric methods partially address this but rely on external frontier models or ground-truth answers, and update rubrics only at coarse granularity. We propose EvoRubrics, a co-evolutionary RL framework where a Policy LLM and a Rubric Generator jointly improve through adversarial interaction within each training step. As the policy improves under the rubric generator's guidance, the rubric generator adapts its criteria to remain discriminative and informative, enabling evaluation to track the policy in real time and naturally inducing an automatic curriculum. Experiments show that EvoRubrics consistently outperforms static and dynamic rubric baselines across benchmarks. The learned Rubric Generator further generalizes as a transferable reward model. Notably, even a fully self-supervised variant without any external supervision achieves meaningful gains, suggesting that co-evolution between generation and evaluation alone can provide sufficiently rich learning signals. Our code is publicly available at https://anonymous.4open.science/r/EvoRubrics-2155/.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• Co-Evolving LLM Evaluators and Policies via DynamicRubric [67.2962847914162]
  Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models.<n>We propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework.<n>In experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines.
  arXiv  Detail & Related papers  (2026-07-22T12:35:40Z)
• LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL [64.4276583943816]
  Reinforcement learning for non-verifiable instruction relies on judges with prompt-specific rubrics as reward signals.<n>While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static.<n>This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal.
  arXiv  Detail & Related papers  (2026-07-05T17:05:13Z)
• INFUSER: Influence-Guided Self-Evolution Improves Reasoning [54.101135873140066]
  We introduce an iterative co-training framework with two co-evolving roles.<n>A solver is trained with standard correctness rewards against the generator-provided answers.<n>An 8B INF co-evolving generator outperforms a frozen 32B thinking generator on math and coding.
  arXiv  Detail & Related papers  (2026-06-08T05:40:36Z)
• Self-evolving LLM agents with in-distribution Optimization [60.05867547965365]
  Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments.<n>We propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning.<n>We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance.
  arXiv  Detail & Related papers  (2026-06-05T15:09:52Z)
• EvoDrive: Pareto Evolution for Safety-Critical Autonomous Driving via Self-Improving LLM Agents [59.44069233970625]
  Existing methods usually manage this trade-off with handcrafteds and overlooking underexplored patterns.<n>Here we present EvoDrive, the first automated, LLM-based agentic evolution framework for scenario generation.<n>EvoDrive employs a simulator-grounded actor-critic architecture where a memory-driven actor iteratively proposes improvements to the generators and critics filter out implausible candidates.
  arXiv  Detail & Related papers  (2026-06-02T14:01:23Z)
• EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation [42.93367895323788]
  EvoRubric is a novel single-policy co-evolutionary Reinforcement Learning framework.<n>It eliminates the reliance on static criteria and on external rubric generators.<n>It consistently outperforms traditional static and external-LLM-driven alignment methods.
  arXiv  Detail & Related papers  (2026-05-28T12:28:49Z)
• EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics [86.49781345669436]
  Language models encode substantial evaluative knowledge from pretraining.<n>Current post-training methods rely on external supervision to produce reward signals.<n>We introduce EVOLM, a method that structures a model's evaluative capacity into explicit discriminative rubrics.
  arXiv  Detail & Related papers  (2026-05-05T15:31:00Z)
• Learning to Reason without External Rewards [100.27210579418562]
  Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision.<n>We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data.<n>We propose Intuitor, an RLIF method that uses a model's own confidence, termed self-certainty, as its sole reward signal.
  arXiv  Detail & Related papers  (2025-05-26T07:01:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.