QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

• URL: http://arxiv.org/abs/2606.03968v1
• Date: Tue, 02 Jun 2026 17:53:04 GMT
• Title: QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards
• Abstract summary: We present a framework that co-designs queries and rubrics.<n>QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline.
• Score: 68.15832368034815
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rubric-based RL is a promising route for extending reinforcement learning beyond verifiable rewards, yet existing methods optimize rubrics while treating the query distribution as fixed. We identify a structural bottleneck: rubric quality is constrained by query structure. Open-ended queries yield vague rubrics; naively narrowing them introduces fabricated references that no model can verify, so all responses fail and training receives no reward signal. We present QUBRIC, a framework that co-designs queries and rubrics. Teacher-derived key points ground the rewriting of open-ended queries into scenario-based, evaluable questions. Contrastive rubric generation then turns teacher-policy gaps into query-level criteria, and learnability filtering retains only informative query-rubric pairs for GRPO training. QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline. Trained only on instruction-following data, it further transfers to three held-out benchmarks spanning legal, moral, and narrative reasoning (+6.3 points on average), with improvements concentrated in reasoning-related dimensions. These results provide evidence that co-designing queries and rubrics can make rubric-based RL a practical complement to RLVR beyond strictly verifiable tasks.

Related papers

• GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation [74.44235943118994]
  GenRubric is a self-evolving framework that improves rubric generation from unlabeled queries.<n>We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality.<n> Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics.
  arXiv  Detail & Related papers  (2026-08-30T15:39:39Z)
• V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning [59.47823383593251]
  Vision-sampling models can produce fluent answers that are insufficiently grounded in the visual evidence.<n>We introduce Visuals-Based Reinforcement Learning, which decomposes reference responses into atomic propositions.<n>We train our model based on the same SFT checkpoint using component-wise, prefix-localized rubric credit.
  arXiv  Detail & Related papers  (2026-08-26T09:40:46Z)
• DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents [17.420077157633852]
  Reinforcement learning with rubric-based rewards improves deep research agents by optimizing them against checkable criteria that translate report quality into reward signals.<n>Most existing studies ask an LLM to generate rubrics for a given query, but when the model fails to infer the underlying information needs, the generated rubrics may be incomplete and reduce RL efficiency.<n>To obtain more reliable query--rubric supervision, we introduce DeepRubric, a data construction framework that reverses this process.
  arXiv  Detail & Related papers  (2026-06-15T17:52:27Z)
• LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards [53.339700196282905]
  Reinforcement learning with verifiable rewards (RLVR) has shown promise for this task.<n>Existing methods are limited by low-confusability distractors and sparse, outcome-only reward signals.<n>textscLongTraceRL produces training contexts that are far more challenging than those built by random sampling or one-shot search.
  arXiv  Detail & Related papers  (2026-05-29T17:51:40Z)
• ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning [71.25239709712193]
  We propose ARES, a framework for automatically constructing rubric-based RL data at scale.<n> ARES conditions generation on domain labels and persona information, and applies validation filters for question self-containment, answer faithfulness, and validity.<n>Experiments on seven benchmarks show that rubric-based trained with ARES, outperforms continual pretraining, supervised fine-tuning, and binary-reward RL, with the largest gains on multi-dimensional open-ended tasks such as healthcare and instruction following.
  arXiv  Detail & Related papers  (2026-05-22T10:09:28Z)
• MDER-DR: Multi-Hop Question Answering with Entity-Centric Summaries [2.756584457554517]
  Map-Disambiguate-Enrich-Reduce (MDER) generates context-derived triple descriptions and integrates them with entity-level summaries.<n>Decompose-Resolve (DR) decomposes user queries into resolvable triples and grounds them in the Knowledge Graph.<n>MDER and DR form an LLM-driven QA pipeline that is robust to sparse, incomplete, and complex relational data.
  arXiv  Detail & Related papers  (2026-03-11T18:38:44Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Decomposed Reasoning with Reinforcement Learning for Relevance Assessment in UGC Platforms [30.51899823655511]
  Retrieval-augmented generation (RAG) plays a critical role in user-generated content platforms.<n> platforms present unique challenges: 1) ambiguous user intent due to sparse user feedback in RAG scenarios, and 2) substantial noise introduced by informal and unstructured language.
  arXiv  Detail & Related papers  (2025-08-04T15:14:09Z)
• Checklists Are Better Than Reward Models For Aligning Language Models [99.1896531064102]
  We propose "Reinforcement Learning from Checklist Feedback" (RLCF)<n>From instructions, we extract checklists and evaluate how well responses satisfy each item.<n>Using both AI judges and specialized verifier programs, we combine these scores to compute rewards for RL.
  arXiv  Detail & Related papers  (2025-07-24T17:58:00Z)
• W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering [28.79851078451609]
  We propose W-RAG, a method that draws weak training signals from the downstream task and fine-tunes the retriever to prioritize passages that most benefit the task.<n>We conduct comprehensive experiments across four publicly available OpenQA datasets to demonstrate that our approach enhances both retrieval and OpenQA performance.
  arXiv  Detail & Related papers  (2024-08-15T22:34:44Z)
• Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning [43.123290672073814]
  We deal with systems that generate summaries from document(s) based on a query. Motivated by the insight that Reinforcement Learning (RL) provides a generalization to Supervised Learning (SL) for Natural Language Generation, we use an RL-based approach for this task. We develop multiple Policy Gradient networks, trained on various reward signals: ROUGE, BLEU, and Semantic Similarity.
  arXiv  Detail & Related papers  (2023-11-29T10:38:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.