Reinforcement Learning with Robust Rubric Rewards

• URL: http://arxiv.org/abs/2605.30244v1
• Date: Thu, 28 May 2026 17:11:03 GMT
• Title: Reinforcement Learning with Robust Rubric Rewards
• Abstract summary: We propose Reinforcement Learning with Robust Rewards ($textRLR3$), extending RLVR from task-level verification to criterion-level verification.<n>$textRLR3$ routes instance-specific rubrics through two execution paths: an LLM-as-an-extractor paired with a deterministic verifier, or an LLM-as-a-Judge for non-verifiable criteria.<n>$textRLR3$ consistently outperforms RLVR, yielding a 4.7-point improvement over the base model and exceeding the official instruct-to-thinking model gap.
• Score: 30.826907231502663
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Reinforcement Learning with Verifiable Rewards (RLVR) is effective for deterministically checkable tasks, many vision-language tasks are partially verifiable, demanding multi-criteria supervision (e.g., perceptual details, reasoning steps, and constraints). Rubrics provide a natural interface for this fine-grained supervision, but their effectiveness depends on the execution accuracy during online RL. We propose Reinforcement Learning with Robust Rubric Rewards ($\text{RLR}^3$), extending RLVR from task-level verification to criterion-level verification. $\text{RLR}^3$ routes instance-specific rubrics through two execution paths: an LLM-as-an-extractor paired with a deterministic verifier, or an LLM-as-a-Judge for non-verifiable criteria. To ensure faithful scoring, $\text{RLR}^3$ introduce a minimal exposure strategy that masks ground truths from extractors and images from judges. Furthermore, $\text{RLR}^3$ employs hierarchical aggregation to prioritize essential criteria over additional criteria, and mitigates score saturation within rollout groups. Evaluated on Qwen3-VL-30B-A3B across 15 benchmarks, $\text{RLR}^3$ consistently outperforms RLVR, yielding a 4.7-point improvement over the base model and exceeding the official instruct-to-thinking model gap. Controlled audits confirm our deterministic verification and minimal exposure significantly reduce exploitable false positives.

Related papers

• RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning [31.35146691876858]
  We propose $textbfRISE-RL$ (Rubric-Informed Selective Exploration), which uses repeatedly missed rubric criteria to elicit privileged trajectories.<n>RISE-RL retains only trajectories whose complete-rubric reward exceeds the mean reward of natural rollouts.<n>It improves the average score by 1.3 points at the 4B scale and $textbf3.3 points at the 14B scale$, including a $textbf6.0-point$ gain on CreativeWriting-V3.
  arXiv  Detail & Related papers  (2026-08-10T05:02:28Z)
• When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion [0.0]
  Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling.<n>We study this pass@k inversion: after training, the policy may solve fewer distinct problems than its base model at large $k$.<n>The failure concentrates on boundary prompts, where the base model contains rare correct trajectories that are recoverable by sampling but too sparse to reliably appear in finite RLVR rollout groups.
  arXiv  Detail & Related papers  (2026-07-12T17:17:18Z)
• IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage [12.520216206687044]
  We present IRDS (Inter- pretable RLVR Data Selection), which selects RLVR training instances on a sparse autoen- coder (SAE) cluster basis.<n>Experiments on three instruction- tuned models and six math reasoning bench- marks show that IRDS achieves the highest overall accuracy.
  arXiv  Detail & Related papers  (2026-05-27T09:58:05Z)
• What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time [57.533031432715084]
  Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams.<n>Existing TTRL methods rely exclusively on positive pseudo-labeling strategies.<n>We propose SCRL, a robust test-time reinforcement learning framework that effectively mitigates label noise amplification.
  arXiv  Detail & Related papers  (2026-03-20T11:47:12Z)
• $\ extbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving [75.29519604607111]
  Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning performance of large language models (LLMs)<n>We show that when the initial direction or quality of the chain-of-thought (CoT) is suboptimal, the model often fails to reach the correct answer.<n>We introduce Reinforcement Learning with Re-solving (Re$2$), in which LLMs learn to flexibly abandon unproductive reasoning paths and restart the solution process when necessary.
  arXiv  Detail & Related papers  (2026-03-07T13:17:46Z)
• ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RL [64.77036363086519]
  We propose ContextRL, a novel framework that leverages context augmentation to overcome these bottlenecks.<n>We provide the reward model with full reference solutions as context, enabling fine-grained process verification to filter out false positives.<n>We also introduce a multi-turn sampling strategy where the reward model generates mistake reports for failed attempts, guiding the policy to "recover" correct responses from previously all-negative groups.
  arXiv  Detail & Related papers  (2026-02-26T04:55:57Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• RLPR: Extrapolating RLVR to General Domains without Verifiers [103.14103272635893]
  We propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains.<n>We find that addressing the high variance of this noisy probability reward is crucial to make it work.<n>RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models.
  arXiv  Detail & Related papers  (2025-06-23T02:56:36Z)
• Direct Reasoning Optimization: LLMs Can Reward And Refine Their Own Reasoning for Open-Ended Tasks [6.881699020319577]
  We propose Direct Reasoning Optimization (DRO), a reinforcement learning framework for fine-tuning Large Language Models (LLMs)<n>DRO is guided by a new reward signal: the Reasoning Reflection Reward (R3)<n>DRO consistently outperforms strong baselines while remaining broadly applicable across both open-ended and structured domains.
  arXiv  Detail & Related papers  (2025-06-16T10:43:38Z)
• R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning [60.17074283370798]
  Retrieval-Augmented Generation (RAG) integrates external knowledge with Large Language Models (LLMs) to enhance factual correctness and hallucination.<n>We propose $textbfR3-RAG$, which uses $textbfR$einforcement learning to make the LLM learn how to $textbfR$eason and $textbfR$etrieve step by step, thus retrieving comprehensive external knowledge and leading to correct answers.
  arXiv  Detail & Related papers  (2025-05-26T12:25:37Z)
• Reinforced Latent Reasoning for LLM-based Recommendation [92.56166822197919]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities in complex problem-solving tasks.<n>Existing methods typically rely on fine-tuning with explicit chain-of-thought (CoT) data.<n>In this work, we explore an alternative approach that shifts from explicit CoT reasoning to compact, information-dense latent reasoning.
  arXiv  Detail & Related papers  (2025-05-25T11:03:45Z)
• LLMSR@XLLM25: Less is More: Enhancing Structured Multi-Agent Reasoning via Quality-Guided Distillation [6.920352059545929]
  We present Less is More, the third-place winning approach in the LLMSR@XLLM25 structural reasoning task.<n>Our approach leverages a multi-agent framework with reverse-prompt induction, retrieval-augmented reasoning synthesis via GPT-4o, and dual-stage reward-guided filtering.<n>All modules are fine-tuned from Meta-Llama-3-8B-Instruct under a unified LoRA+ setup.
  arXiv  Detail & Related papers  (2025-04-23T04:19:52Z)
• Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning [54.585428241509234]
  We propose R$3$: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL) RL employs only outcome supervision to achieve the benefits of process supervision for large language models.
  arXiv  Detail & Related papers  (2024-02-08T16:46:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.