SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

• URL: http://arxiv.org/abs/2607.26873v2
• Date: Fri, 31 Jul 2026 13:09:25 GMT
• Title: SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
• Abstract summary: Test-time reinforcement learning (TTRL) enables language models to self-evolve at inference time without labeled feedback.<n>We introduce SERPO, which replaces answer voting with a closed loop that co-evolves response evidence, querytokens, and policy parameters.<n>New actor rollouts then refresh both the archives and rubrics, closing the three-way evolution loop.
• Score: 14.505575259468053
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Test-time reinforcement learning (TTRL) enables language models to self-evolve at inference time without labeled feedback. Existing methods rely on answer voting and therefore do not extend naturally to open-ended generation, where valid responses cannot be mapped to a shared canonical answer. Without external reward models or stronger judges, adaptation must instead construct reliable rewards from the model's own outputs. We introduce SERPO (Self-Evolving Rubric Policy Optimization), which replaces answer voting with a closed loop that co-evolves response evidence, query-specific rubrics, and policy parameters. Good-Normal-Bad (G-N-B) response evolution organizes maximally separated rollouts into ordered archives; rubric evolution retains criteria that discriminate these archives; probabilistic criterion scoring converts verdict-token likelihoods into reward signals; and policy evolution optimizes the actor with the resulting signals. New actor rollouts then refresh both the archives and rubrics, closing the three-way evolution loop. Across two model configurations, two in-domain benchmarks, and four OOD benchmarks, SERPO improves HealthBench and ResearchQA by up to 20.63 and 20.31 points over the corresponding base models, raises the six-benchmark macro-average by up to 8.06 points, and supports OOD transfer and continued cross-benchmark evolution.

Related papers

• Co-Evolving LLM Evaluators and Policies via DynamicRubric [67.2962847914162]
  Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models.<n>We propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework.<n>In experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines.
  arXiv  Detail & Related papers  (2026-07-22T12:35:40Z)
• The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators [17.44053709945519]
  Self-improving agents are state-of-the-art (SOTA) on agentic coding benchmarks and have recently been extended to general domains.<n>We introduce the Red Queen Godel Machine (RQGM), an evolutionary framework for self-improvement under non-stationary utilities.<n>We show that the RQGM improves test pass rate over the prior SOTA by adding a complementary agent-as-a-judge code-review signal.<n>We then turn to scientific paper writing and reviewing, and Olympiad-level proof writing and grading, where the RQGM improves performance over prior self-improving agents
  arXiv  Detail & Related papers  (2026-06-24T18:38:26Z)
• EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning [22.209572966965165]
  EvoRubrics is a co-evolutionary framework where a Policy Generator and a RL Generator jointly improve through adversarial interaction within each training step.<n>EvoRubrics consistently outperforms static and dynamic rubrics across benchmarks.
  arXiv  Detail & Related papers  (2026-06-22T08:46:23Z)
• INFUSER: Influence-Guided Self-Evolution Improves Reasoning [54.101135873140066]
  We introduce an iterative co-training framework with two co-evolving roles.<n>A solver is trained with standard correctness rewards against the generator-provided answers.<n>An 8B INF co-evolving generator outperforms a frozen 32B thinking generator on math and coding.
  arXiv  Detail & Related papers  (2026-06-08T05:40:36Z)
• From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding [35.008790179255136]
  We propose a group-revision optimisation paradigm that enhances learning on hard cases.<n>Inspired by reward shaping, we introduce a consolidation process that quantifies each candidate's improvement over the initial attempt.<n>Our method achieves consistent gains across referring and reasoning segmentation, REC, and counting benchmarks compared with prior GRPO-based models.
  arXiv  Detail & Related papers  (2026-05-15T13:41:41Z)
• EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics [86.49781345669436]
  Language models encode substantial evaluative knowledge from pretraining.<n>Current post-training methods rely on external supervision to produce reward signals.<n>We introduce EVOLM, a method that structures a model's evaluative capacity into explicit discriminative rubrics.
  arXiv  Detail & Related papers  (2026-05-05T15:31:00Z)
• iGRPO: Self-Feedback-Driven LLM Reasoning [88.83313431248473]
  Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions.<n>We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts.<n>Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models.
  arXiv  Detail & Related papers  (2026-02-09T18:45:11Z)
• GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning [53.894789613838654]
  We introduce SEED-Bench-R1, a benchmark with complex real-world videos requiring balanced perception and reasoning.<n>Using SEED-Bench-R1, we find that standard GRPO, while improving answer accuracy, often reduces logical coherence between reasoning steps and answers, with only a 57.9% consistency rate.<n>We propose GRPO-CARE, a consistency-aware RL framework optimizing both answer correctness and reasoning coherence without explicit supervision.
  arXiv  Detail & Related papers  (2025-06-19T08:49:13Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.