Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence

• URL: http://arxiv.org/abs/2607.15092v1
• Date: Thu, 16 Jul 2026 15:01:33 GMT
• Title: Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence
• Abstract summary: Existing approaches often derive supervision from human-written rubrics, preference data, or sampled responses.<n>We introduce a query-only framework that evolves a rubric set from an empty set without external annotations or model training.<n>Experiments across five benchmark suites demonstrate the effectiveness of preferences on Trial.
• Score: 21.413257942637888
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rubrics provide structured, fine-grained signals for training and evaluating large language models (LLMs). Yet reliable query-specific rubrics are difficult to construct. Existing approaches often derive supervision from human-written rubrics, preference data, or sampled responses. Direct query-to-rubric generation avoids these resources, but provides no explicit check that a plausible rubric is useful. Such a rubric may fail to distinguish answer quality, reward an optional style, or penalize a valid alternative strategy. We introduce Rubrics on Trial, a query-only framework that evolves a rubric set from an empty set without external annotations or model training. It derives supervision solely from synthetic rubric-conditioned response pairs and validates each proposed rubric before adding it, screening out non-discriminative, over-specific, and style-only candidate rubrics. Experiments across five preference benchmark suites demonstrate the effectiveness of Rubrics on Trial, which achieves the best average accuracy and leads on six of seven evaluation sets.

Related papers

• GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation [74.44235943118994]
  GenRubric is a self-evolving framework that improves rubric generation from unlabeled queries.<n>We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality.<n> Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics.
  arXiv  Detail & Related papers  (2026-08-30T15:39:39Z)
• APTER: Adaptive Post-Training with Expert-Grounded Rubrics [23.84845758379717]
  APTER is a framework that integrates structured domain knowledge into fine-grained evaluation, optimization, and diagnosis for specialized complex reasoning.<n>APTER improves the mathematics and medical averages over the corresponding base models by up to 15.86 and 8.04 points, respectively.
  arXiv  Detail & Related papers  (2026-08-14T11:42:29Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• Provably Efficient Personalized Multi-Objective Bandits with Proactive Conversational Queries [24.496126767935937]
  We formalize a proactive query-based framework in which user queries provide structured preference signals.<n>We prove that proactive queries accelerate preference estimation and yield improved regret scaling over prior preference-aware MO-MAB methods.
  arXiv  Detail & Related papers  (2026-06-07T02:19:59Z)
• QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards [68.15832368034815]
  We present a framework that co-designs queries and rubrics.<n>QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline.
  arXiv  Detail & Related papers  (2026-06-02T17:53:04Z)
• TopBench: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering [80.93487993878836]
  A common class of real-world queries is implicitly predictive, requiring the inference of unobserved answers from historical patterns rather than mere retrieval.<n>These queries introduce two challenges: recognizing latent intent and reliable predictive reasoning over massive tables.<n>We introduce TopBench, a benchmark consisting of 779 samples across four sub-tasks, ranging from single-point prediction to decision making, treatment effect analysis, and complex reasoning.
  arXiv  Detail & Related papers  (2026-04-30T16:22:51Z)
• RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [11.21565372620296]
  Large language models (LLMs) are increasingly evaluated and sometimes trained using automated graders that output scalar scores or preferences.<n>This lack of interpretability limits their usefulness for model development, dataset curation, and high-stakes deployment.<n>We introduce a simple strategy,RAG, which retrieves domain knowledge via rubrics at inference time from related queries.
  arXiv  Detail & Related papers  (2026-03-21T17:10:14Z)
• Hypothesis-Conditioned Query Rewriting for Decision-Useful Retrieval [39.146761527401424]
  Retrieval-Augmented Generation (RAG) improves Large Language Models (LLMs) by grounding generation in external, non-parametric knowledge.<n>Existing RAG methods rely on a single initial query, which often favors topical relevance over decision-relevant evidence.<n>We propose Hypothesis-Conditioned Query Rewriting (HCQR), a training-free pre-retrieval framework that reorients RAG from topic-oriented retrieval to evidence-oriented retrieval.
  arXiv  Detail & Related papers  (2026-03-19T15:15:58Z)
• Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [80.12435680651488]
  We propose a pipeline to train human-preference-aligned query-specific rubric generators tailored for DeepResearch report generation.<n>We first construct a dataset of DeepResearch-style annotated queries with human preferences over paired reports, and train rubric generators via reinforcement learning.<n>We empirically show that our proposed rubric generators deliver more discriminative and better human-aligned supervision than existing rubric design strategies.
  arXiv  Detail & Related papers  (2026-02-03T15:09:56Z)
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [38.1645520104553]
  We introduce OpenRubrics, a large-scale collection of (prompt,explicit) pairs for training rubric-generation and rubric-based reward models.<n>To elicit discriminative and comprehensive evaluation signals, we introduce Contrastive Generation (CRG), which derives both hard rules (implicit qualities) and principles (implicit qualities) by contrasting preferred and rejected responses.<n>Our results show that rubrics provide scalable alignment signals that narrow the gap between costly human evaluation and automated reward modeling.
  arXiv  Detail & Related papers  (2025-10-09T03:31:26Z)
• Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries [85.81295563405433]
  We present a protocol that synthetically constructs context surrounding an under-specified query and provides it during evaluation.<n>We find that the presence of context can 1) alter conclusions drawn from evaluation, even flipping benchmark rankings between model pairs, 2) nudge evaluators to make fewer judgments based on surface-level criteria, like style, and 3) provide new insights about model behavior across diverse contexts.
  arXiv  Detail & Related papers  (2024-11-11T18:58:38Z)
• A Semantic-based Method for Unsupervised Commonsense Question Answering [40.18557352036813]
  Unsupervised commonsense question answering is appealing since it does not rely on any labeled task data. We present a novel SEmantic-based Question Answering method (SEQA) for unsupervised commonsense question answering.
  arXiv  Detail & Related papers  (2021-05-31T08:21:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.