GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

• URL: http://arxiv.org/abs/2608.29856v1
• Date: Sun, 30 Aug 2026 15:39:39 GMT
• Title: GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
• Abstract summary: GenRubric is a self-evolving framework that improves rubric generation from unlabeled queries.<n>We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality.<n> Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics.
• Score: 74.44235943118994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are increasingly used as scalable evaluators for open-ended tasks. However, many LLM judges derive query-specific criteria during scoring, leaving the evaluation requirements insufficiently specified and their coverage difficult to audit. Query-specific rubrics make these requirements explicit, but expert-written rubrics are costly to construct, while existing automatic methods typically rely on inference-time refinement or external supervision. We introduce GenRubric, a self-evolving framework that improves rubric generation from unlabeled queries without requiring additional human annotations during self-evolution. Our approach is based on rubric-induced self-consistency: independently sampled rubrics for the same query provide partial views of its latent evaluation requirements, and a comprehensive rubric should induce a response that generalizes across these complementary evaluation views. We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality. We train GenRubric models at 4B, 8B, and 14B scales across multiple domains. Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics. The improvements further generalize to held-out domains, demonstrating the potential of self-evolving rubric generation for scalable and query-specific LLM evaluation. Code and models are publicly available at https://github.com/foggpoy/GenRubric.

Related papers

• APTER: Adaptive Post-Training with Expert-Grounded Rubrics [23.84845758379717]
  APTER is a framework that integrates structured domain knowledge into fine-grained evaluation, optimization, and diagnosis for specialized complex reasoning.<n>APTER improves the mathematics and medical averages over the corresponding base models by up to 15.86 and 8.04 points, respectively.
  arXiv  Detail & Related papers  (2026-08-14T11:42:29Z)
• Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling [39.325422496065435]
  Multi-Role Generation (MRRG) is a training-free and reference-free framework that elicits evaluation criteria from multiple roles.<n>MRRG can be used both to validate pairwise preferences and to provide rewards for RLO-style Reinforcement Learning with Verifiable Rewards (RLVR)
  arXiv  Detail & Related papers  (2026-07-02T07:50:38Z)
• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [11.21565372620296]
  Large language models (LLMs) are increasingly evaluated and sometimes trained using automated graders that output scalar scores or preferences.<n>This lack of interpretability limits their usefulness for model development, dataset curation, and high-stakes deployment.<n>We introduce a simple strategy,RAG, which retrieves domain knowledge via rubrics at inference time from related queries.
  arXiv  Detail & Related papers  (2026-03-21T17:10:14Z)
• Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents [81.79110139097297]
  General-purpose embedding models excel at recognizing semantic similarities but fail to capture the characteristics of texts specified by user instructions.<n>We reframe instruction-following clustering as a generative task and train large reasoning models as autonomous clustering agents.<n>Our reasoning-driven training pipeline enables LRMs to interpret high-level clustering instructions and then infer the corresponding latent groupings.
  arXiv  Detail & Related papers  (2026-03-06T10:15:54Z)
• Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [80.12435680651488]
  We propose a pipeline to train human-preference-aligned query-specific rubric generators tailored for DeepResearch report generation.<n>We first construct a dataset of DeepResearch-style annotated queries with human preferences over paired reports, and train rubric generators via reinforcement learning.<n>We empirically show that our proposed rubric generators deliver more discriminative and better human-aligned supervision than existing rubric design strategies.
  arXiv  Detail & Related papers  (2026-02-03T15:09:56Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• SedarEval: Automated Evaluation using Self-Adaptive Rubrics [4.97150240417381]
  We propose a new evaluation paradigm based on self-adaptive rubrics.<n>SedarEval consists of 1,000 meticulously crafted questions, each with its own self-adaptive rubric.<n>We train a specialized evaluator language model (evaluator LM) to supplant human graders.
  arXiv  Detail & Related papers  (2025-01-26T16:45:09Z)
• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models [94.31327813151208]
  BiGGen Bench is a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks.<n>A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation.
  arXiv  Detail & Related papers  (2024-06-09T12:30:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.