Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling

• URL: http://arxiv.org/abs/2607.01830v1
• Date: Thu, 02 Jul 2026 07:50:38 GMT
• Title: Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
• Abstract summary: Multi-Role Generation (MRRG) is a training-free and reference-free framework that elicits evaluation criteria from multiple roles.<n>MRRG can be used both to validate pairwise preferences and to provide rewards for RLO-style Reinforcement Learning with Verifiable Rewards (RLVR)
• Score: 39.325422496065435
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks. Rubric-based judges offer a transparent way to decompose such judgments into explicit evaluation criteria, but existing annotation-free rubric generators typically rely on a single generic evaluator. As a result, they may overlook important dimensions of human preference, a failure mode we term dimensional blind spots. To address this limitation, we propose Multi-Role Rubric Generation (MRRG), a training-free and reference-free framework that elicits evaluation criteria from multiple complementary roles and consolidates them into an auditable rubric-based scorer. This scorer can be used both to validate pairwise preferences and to provide rewards for GRPO-style Reinforcement Learning with Verifiable Rewards (RLVR). Experiments on preference validation benchmarks show that MRRG consistently outperforms single-role rubric generation baselines across multiple backbone models. Further RLVR experiments demonstrate that MRRG yields a stronger reward signal for improving open-ended generation.

Related papers

• GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation [74.44235943118994]
  GenRubric is a self-evolving framework that improves rubric generation from unlabeled queries.<n>We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality.<n> Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics.
  arXiv  Detail & Related papers  (2026-08-30T15:39:39Z)
• Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria [17.272139541614383]
  Auto-Rubric as Reward (ARR) is a framework that reframes reward modeling from implicit weight optimization to explicit, criteria-based decomposition.<n>ARR externalizes a VLM's preference knowledge as prompt-specific gradients, translating holistic intent into independently verifiable quality dimensions.<n>ARR-RPO outperforms pairwise reward models and VLM judges on text-to-image generation and image editing benchmarks.
  arXiv  Detail & Related papers  (2026-05-08T18:05:27Z)
• RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time [51.256131853751754]
  We show that teaching reward models to produce explicit, multi-dimensional critiques transforms them from passive evaluators into active optimization tools.<n>We introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data.<n>The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models.
  arXiv  Detail & Related papers  (2026-04-13T15:38:09Z)
• CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [61.75914342638658]
  CDRRM (Contrast-Driven Reward Model) is a framework for high-quality rubric generation and guided preference judgment.<n>This work offers a scalable, interpretable, and data-efficient path for reward modeling.
  arXiv  Detail & Related papers  (2026-03-09T07:15:23Z)
• From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning [7.6602542594279335]
  We propose Reinforcement Learning with Relative Rewards to shift reward shaping from absolute scoring to relative ranking.<n>We show that RLRR yields consistent performance improvements over standard group-based baselines across reasoning benchmarks and open-ended generation tasks.
  arXiv  Detail & Related papers  (2026-01-30T15:07:06Z)
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [38.1645520104553]
  We introduce OpenRubrics, a large-scale collection of (prompt,explicit) pairs for training rubric-generation and rubric-based reward models.<n>To elicit discriminative and comprehensive evaluation signals, we introduce Contrastive Generation (CRG), which derives both hard rules (implicit qualities) and principles (implicit qualities) by contrasting preferred and rejected responses.<n>Our results show that rubrics provide scalable alignment signals that narrow the gap between costly human evaluation and automated reward modeling.
  arXiv  Detail & Related papers  (2025-10-09T03:31:26Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)
• Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards [11.149294285483782]
  We propose a unified RLVR-based training paradigm that bridges the gap between non-verifiable tasks and verifiable rewards.<n>We introduce a writing-principle-based pairwise Generative Reward Model (GenRM) and a novel Bootstrapped Relative Policy Optimization (BRPO) algorithm.<n>Our approach empowers LLMs to develop robust writing capabilities without supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-05-30T14:34:57Z)
• RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning [64.46921169261852]
  RAG-Zeval is a novel end-to-end framework that formulates faithfulness and correctness evaluation as a rule-guided reasoning task.<n>Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments.<n>Experiments demonstrate RAG-Zeval's superior performance, achieving the strongest correlation with human judgments.
  arXiv  Detail & Related papers  (2025-05-28T14:55:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.