EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation

• URL: http://arxiv.org/abs/2605.29847v1
• Date: Thu, 28 May 2026 12:28:49 GMT
• Title: EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation
• Abstract summary: EvoRubric is a novel single-policy co-evolutionary Reinforcement Learning framework.<n>It eliminates the reliance on static criteria and on external rubric generators.<n>It consistently outperforms traditional static and external-LLM-driven alignment methods.
• Score: 42.93367895323788
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning (RL) has significantly advanced Large Language Models (LLMs) in verifiable domains, but aligning models for open-ended generation remains profoundly challenging due to the lack of definitive rewards. Current rubric-based RL methods mitigate this by employing explicit criteria; however, they rely heavily on static, human-annotated rubrics that inevitably cause policy lag, or expensive external proprietary models for dynamic updates. In this paper, we propose EvoRubric, a novel single-policy co-evolutionary RL framework that eliminates the reliance on static criteria and on external rubric generators. By unifying response generation and rubric generation under a single parameterized policy, EvoRubric dynamically alternates between a Reasoner and a Rubric Generator. To prevent reward hacking and ensure the reliability of generated signals, we introduce a multi-level verification pipeline featuring a meta-verifier, zero-variance pruning, and a Leave-One-Out peer consensus mechanism. Validated criteria are dynamically archived into a memory pool, yielding dense, multi-objective rewards to continuously co-optimize both roles. Extensive experiments across Medical, Writing, and Science domains demonstrate that EvoRubric consistently outperforms traditional static and external-LLM-driven alignment methods. Notably, our framework is compatible with human-expert priors. When initialized with expert-annotated rubrics, EvoRubric can further uncover novel, discriminative dimensions, achieving better performance than relying solely on static expert annotations.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning [22.209572966965165]
  EvoRubrics is a co-evolutionary framework where a Policy Generator and a RL Generator jointly improve through adversarial interaction within each training step.<n>EvoRubrics consistently outperforms static and dynamic rubrics across benchmarks.
  arXiv  Detail & Related papers  (2026-06-22T08:46:23Z)
• EvoDrive: Pareto Evolution for Safety-Critical Autonomous Driving via Self-Improving LLM Agents [59.44069233970625]
  Existing methods usually manage this trade-off with handcrafteds and overlooking underexplored patterns.<n>Here we present EvoDrive, the first automated, LLM-based agentic evolution framework for scenario generation.<n>EvoDrive employs a simulator-grounded actor-critic architecture where a memory-driven actor iteratively proposes improvements to the generators and critics filter out implausible candidates.
  arXiv  Detail & Related papers  (2026-06-02T14:01:23Z)
• AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution [61.593935260052795]
  Post-training viaSupervised Fine-Tuning (SFT) and Reinforcement Learning (RL) is crucial for enhancing reasoning in Multimodal Large Language Models (MLLMs)<n>Existing paradigms often reach a performance bottleneck due to the limitations of static data.<n>We propose Anchor Evolution (AnE), a new paradigm that integrates truth-anchored data curation and model evolution.
  arXiv  Detail & Related papers  (2026-05-25T08:26:34Z)
• Reasoning as an Attack Surface: Adaptive Evolutionary CoT Jailbreaks for LLMs [56.53007443197966]
  Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in reasoning and generation tasks.<n>Their explicit chain-of-thought (CoT) mechanism introduces new security risks, making them particularly vulnerable to jailbreak attacks.<n>We propose an adaptive evolutionary CoT jailbreak framework, called AE-CoT, to overcome these limitations.
  arXiv  Detail & Related papers  (2026-05-23T10:11:32Z)
• AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward [52.7128440950672]
  We propose AlphaGRPO to enhance multimodal generation capabilities without an additional cold-start stage.<n>Our approach unlocks the model's intrinsic potential to perform advanced reasoning tasks.<n>To address the challenge of providing stable supervision for real-world multimodal generation, we introduce the Decompositional Verifiable Reward (DVReward)
  arXiv  Detail & Related papers  (2026-05-12T17:59:47Z)
• Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria [17.272139541614383]
  Auto-Rubric as Reward (ARR) is a framework that reframes reward modeling from implicit weight optimization to explicit, criteria-based decomposition.<n>ARR externalizes a VLM's preference knowledge as prompt-specific gradients, translating holistic intent into independently verifiable quality dimensions.<n>ARR-RPO outperforms pairwise reward models and VLM judges on text-to-image generation and image editing benchmarks.
  arXiv  Detail & Related papers  (2026-05-08T18:05:27Z)
• EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning [15.169565493841384]
  EvoIdeator is a framework that facilitates the evolution of scientific ideas by aligning the RL training objective with textbfchecklist-grounded feedback<n>EvoIdeator, built on Qwen3-4B, significantly outperforms much larger frontier models across key scientific metrics.
  arXiv  Detail & Related papers  (2026-03-23T09:15:26Z)
• R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging [69.96389360650072]
  We show that reasoning fidelity is highly predictive of downstream RLHF outcomes, beyond standard label accuracy.<n>We propose Rationale-Centric Alignment, R-Align, which augments training with gold judgments and explicitly supervises rationale alignment.
  arXiv  Detail & Related papers  (2026-02-06T15:17:11Z)
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [38.1645520104553]
  We introduce OpenRubrics, a large-scale collection of (prompt,explicit) pairs for training rubric-generation and rubric-based reward models.<n>To elicit discriminative and comprehensive evaluation signals, we introduce Contrastive Generation (CRG), which derives both hard rules (implicit qualities) and principles (implicit qualities) by contrasting preferred and rejected responses.<n>Our results show that rubrics provide scalable alignment signals that narrow the gap between costly human evaluation and automated reward modeling.
  arXiv  Detail & Related papers  (2025-10-09T03:31:26Z)
• STARec: An Efficient Agent Framework for Recommender Systems via Autonomous Deliberate Reasoning [54.28691219536054]
  We introduce STARec, a slow-thinking augmented agent framework that endows recommender systems with autonomous deliberative reasoning capabilities.<n>We develop anchored reinforcement training - a two-stage paradigm combining structured knowledge distillation from advanced reasoning models with preference-aligned reward shaping.<n>Experiments on MovieLens 1M and Amazon CDs benchmarks demonstrate that STARec achieves substantial performance gains compared with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-08-26T08:47:58Z)
• Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards [11.149294285483782]
  We propose a unified RLVR-based training paradigm that bridges the gap between non-verifiable tasks and verifiable rewards.<n>We introduce a writing-principle-based pairwise Generative Reward Model (GenRM) and a novel Bootstrapped Relative Policy Optimization (BRPO) algorithm.<n>Our approach empowers LLMs to develop robust writing capabilities without supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-05-30T14:34:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.