Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness

• URL: http://arxiv.org/abs/2607.19322v1
• Date: Tue, 21 Jul 2026 17:42:50 GMT
• Title: Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
• Abstract summary: Gamut is a benchmark for factual completeness in long-form generation.<n>We construct 1,813 questions grounded in real wearable imagery across 10 diverse domains.
• Score: 8.98920719663984
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating the factuality of long-form generations has focused predominantly on precision, measuring whether the claims a model makes are correct. The dominant decompose-search-verify pipeline catches incorrect claims well but says little about whether a response contains all the information it should. Measuring factual completeness, the missing half of factuality, is harder: it requires enumerating the full set of facts a complete answer should contain, and these facts rarely form a flat list. They often involve open-ended sets where coverage is what matters, ordered processes, and relationships among facts that a list of independent boolean checks fails to capture. We introduce a two-level meta-rubric framework for evaluating open-ended generation, and instantiate it as Gamut (Grounded Assessment of Multimodal Factuality), a benchmark for factual completeness in long-form generation. The framework rests on a two-level rubric representation: a structured meta-rubric captures the organization and importance of the required content, which is then mechanically compiled into a flat checklist of binary, machine-gradable rubrics that an LLM judge scores reliably. We construct 1,813 questions grounded in real wearable imagery across 10 diverse domains, each paired with an evidence-backed rubric verified by expert human annotators. Because the framework is modality-agnostic, we also release a text-only variant. Evaluating 14 frontier and open-weight models, we find the benchmark genuinely challenging (best score 58.7% from Gemini 3.1 Pro), highly discriminative, and robust to the choice of judge.

Related papers

• Noēsis: Deterministic-First Retrieval with Two-Tier Context Hydration for Factuality-Critical Queries on Small Local Models [0.0]
  We present Noesis, the deterministic-first query plane of the Noesis architecture.<n>Noesis makes every deterministic judgment before generation.<n>Every reported value is traceable to its exact source and position by construction.
  arXiv  Detail & Related papers  (2026-09-07T15:49:16Z)
• Achieving Text-based Person Retrieval with Any Granularity [80.69427598925718]
  This paper introduces a new paradigm, Text-based Person Retrieval with Any Granularity.<n>We formalize a five-level granularity spectrum and construct UFine6926-MG, a high-quality multi-grained annotated dataset.<n>We then propose the Cross-modal Multi-grained Aligning and Matching framework.
  arXiv  Detail & Related papers  (2026-07-23T08:42:25Z)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle [0.0]
  Reference-free faithfulness metrics verify each atomic claim a model makes against ground truth.<n>We show they share a blind spot: they measure only precision, are the stated claims supported?<n>We make this measurable using Formula 1 telemetry, a domain where strategic ground truth is derived deterministically.
  arXiv  Detail & Related papers  (2026-06-08T11:56:25Z)
• DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark [48.84943754804533]
  Multimodal documents contain diverse elements, such as tables, figures, and layouts.<n>Current approaches typically combine dense visual embedding models with supervised rerankers to achieve high-precision retrieval.<n>We propose DocRetriever, a plug-and-play framework that enhances visual retrieval via a layout-aware sparse embedding technique.
  arXiv  Detail & Related papers  (2026-05-28T14:50:53Z)
• AssertLLM2: A Comprehensive LLM Benchmark for Assertion Generation from Design Specifications [7.797719198801956]
  We introduce AssertLLM2, an open-source benchmark for realistic assertion generation in hardware verification.<n>AssertLLM2 contains 83 real-world designs across 13 functional categories.<n>For each design, the benchmark provides a structured design specification, a verified dependency-complete golden RTL, and systematically mutated buggy RTL variants.
  arXiv  Detail & Related papers  (2026-05-26T10:49:25Z)
• DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding [29.270128057512284]
  We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem.<n>We design a four-stage evaluation protocol that audits each level of the trajectory independently.<n>We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems.
  arXiv  Detail & Related papers  (2026-05-09T11:12:59Z)
• FABLE: Fine-grained Fact Anchoring for Unstructured Model Editing [28.870053452479443]
  Unstructured model editing aims to update models with real-world text, yet existing methods often memorize text holistically without reliable fine-grained fact access.<n>We propose FABLE, a hierarchical framework that decouples fine-grained fact injection from holistic text generation.<n>Experiments show that FABLE substantially improves fine-grained question answering while maintaining state-of-the-art holistic editing performance.
  arXiv  Detail & Related papers  (2026-04-14T10:41:36Z)
• The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality [70.45240108873001]
  The FACTS Leaderboard is an online leaderboard suite that comprehensively evaluates the ability of language models to generate factually accurate text.<n>The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards.
  arXiv  Detail & Related papers  (2025-12-11T16:35:14Z)
• MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval [86.35779264575154]
  Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios.<n>We introduce MR$2$-Bench, a reasoning-intensive benchmark for multimodal retrieval.
  arXiv  Detail & Related papers  (2025-09-30T15:09:14Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.