EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics

• URL: http://arxiv.org/abs/2605.03871v1
• Date: Tue, 05 May 2026 15:31:00 GMT
• Title: EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics
• Abstract summary: Language models encode substantial evaluative knowledge from pretraining.<n>Current post-training methods rely on external supervision to produce reward signals.<n>We introduce EVOLM, a method that structures a model's evaluative capacity into explicit discriminative rubrics.
• Score: 86.49781345669436
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models encode substantial evaluative knowledge from pretraining, yet current post-training methods rely on external supervision (human annotations, proprietary models, or scalar reward models) to produce reward signals. Each imposes a ceiling. Human judgment cannot supervise capabilities beyond its own, proprietary APIs create dependencies, and verifiable rewards cover only domains with ground-truth answers. Self-improvement from a model's own evaluative capacity is a reward source that scales with the model itself, yet remains largely untapped by current methods. We introduce EVOLM, a post-training method that structures this capacity into explicit discriminative rubrics and uses them as training signal. EVOLM trains two capabilities within a single language model in alternation: (1) a rubric generator producing instance-specific evaluation criteria optimized for discriminative utility, which maximizes a small frozen judge's ability to distinguish preferred from dispreferred responses; and (2) a policy trained using those rubric-conditioned scores as reward. All preference signals are constructed from the policy's own outputs via temporal contrast with earlier checkpoints, requiring no human annotation or external supervision. EVOLM trains a Qwen3-8B model to generate rubrics that outperform GPT-4.1 on RewardBench-2 by 25.7%. The co-trained policy achieves 69.3% average on the OLMo3-Adapt suite, outperforming policies trained with GPT-4.1 prompted rubrics by 3.9% and with the state-of-the-art 8B reward model SkyWork-RM by 16%. Overall, EVOLM demonstrates that structuring a model's evaluative capacity into co-evolving discriminative rubrics enables self-improvement without external supervision.

Related papers

• SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning [14.505575259468053]
  Test-time reinforcement learning (TTRL) enables language models to self-evolve at inference time without labeled feedback.<n>We introduce SERPO, which replaces answer voting with a closed loop that co-evolves response evidence, querytokens, and policy parameters.<n>New actor rollouts then refresh both the archives and rubrics, closing the three-way evolution loop.
  arXiv  Detail & Related papers  (2026-07-29T13:03:07Z)
• EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning [22.209572966965165]
  EvoRubrics is a co-evolutionary framework where a Policy Generator and a RL Generator jointly improve through adversarial interaction within each training step.<n>EvoRubrics consistently outperforms static and dynamic rubrics across benchmarks.
  arXiv  Detail & Related papers  (2026-06-22T08:46:23Z)
• Characterizing Model-Native Skills [16.891026204025838]
  Skills are a natural unit for describing what a language model can do and how its behavior can be changed.<n>Existing characterizations rely on human-written, textual descriptions, or manual profiling pipelines.<n>We argue that when the goal is to intervene on model behavior, skill characterization should be *model-native*: grounded in the model's own representations.
  arXiv  Detail & Related papers  (2026-04-19T20:58:25Z)
• LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model [99.71684530652942]
  We show that LLaVA-Critic-R1 emerges as a top-performing critic but also as a competitive policy model.<n>Applying self-critique at test time yields an average +13.8% improvement on five representative reasoning tasks.<n>Our results reveal that RL training on critic data can produce a unified model excelling at both evaluation and generation.
  arXiv  Detail & Related papers  (2025-08-31T03:08:02Z)
• URPO: A Unified Reward & Policy Optimization Framework for Large Language Models [10.511836918064724]
  We propose a novel framework that unifies instruction-following ("player") and reward modeling ("referee") within a single model and a single training phase.<n>Our method recasts all alignment data-including preference pairs, verifiable reasoning, and open-ended instructions-into a unified generative format.<n>Experiments on the Qwen2.5-7B model demonstrate URPO's superiority.
  arXiv  Detail & Related papers  (2025-07-23T13:52:27Z)
• Pre-Trained Policy Discriminators are General Reward Models [81.3974586561645]
  We propose a scalable pre-training method named Policy Discriminative Learning (POLAR)<n>POLAR trains a reward model (RM) to discern identical policies and discriminate different ones.<n> Empirical results show that POLAR substantially outperforms traditional non-pre-trained methods.
  arXiv  Detail & Related papers  (2025-07-07T16:56:31Z)
• Learning to Reason without External Rewards [100.27210579418562]
  Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision.<n>We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data.<n>We propose Intuitor, an RLIF method that uses a model's own confidence, termed self-certainty, as its sole reward signal.
  arXiv  Detail & Related papers  (2025-05-26T07:01:06Z)
• Poor-Supervised Evaluation for SuperLLM via Mutual Consistency [20.138831477848615]
  We propose the PoEM framework to conduct evaluation without accurate labels. We first prove that the capability of a model can be equivalently assessed by the consistency between it and certain reference model. To alleviate the insufficiencies of the conditions in reality, we introduce an algorithm that treats humans (when available) and the models under evaluation as reference models.
  arXiv  Detail & Related papers  (2024-08-25T06:49:03Z)
• Self-Taught Evaluators [77.92610887220594]
  We present an approach that aims to im-proves without human annotations, using synthetic training data only. Our Self-Taught Evaluator can improve a strong LLM from 75.4 to 88.3 on RewardBench.
  arXiv  Detail & Related papers  (2024-08-05T17:57:02Z)
• Self-Play Preference Optimization for Language Model Alignment [75.83359213697854]
  Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences. We propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium.
  arXiv  Detail & Related papers  (2024-05-01T17:59:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.