Eval Factsheets: A Structured Framework for Documenting AI Evaluations

• URL: http://arxiv.org/abs/2512.04062v1
• Date: Wed, 03 Dec 2025 18:46:50 GMT
• Title: Eval Factsheets: A Structured Framework for Documenting AI Evaluations
• Abstract summary: We introduce Eval Factsheets, a framework for documenting AI system evaluations.<n>Our framework organizes evaluation characteristics across five fundamental dimensions.<n>We demonstrate that Eval Factsheets effectively captures diverse evaluation paradigms.
• Score: 18.275882528334794
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid proliferation of benchmarks has created significant challenges in reproducibility, transparency, and informed decision-making. However, unlike datasets and models -- which benefit from structured documentation frameworks like Datasheets and Model Cards -- evaluation methodologies lack systematic documentation standards. We introduce Eval Factsheets, a structured, descriptive framework for documenting AI system evaluations through a comprehensive taxonomy and questionnaire-based approach. Our framework organizes evaluation characteristics across five fundamental dimensions: Context (Who made the evaluation and when?), Scope (What does it evaluate?), Structure (With what the evaluation is built?), Method (How does it work?) and Alignment (In what ways is it reliable/valid/robust?). We implement this taxonomy as a practical questionnaire spanning five sections with mandatory and recommended documentation elements. Through case studies on multiple benchmarks, we demonstrate that Eval Factsheets effectively captures diverse evaluation paradigms -- from traditional benchmarks to LLM-as-judge methodologies -- while maintaining consistency and comparability. We hope Eval Factsheets are incorporated into both existing and newly released evaluation frameworks and lead to more transparency and reproducibility.

Related papers

• Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation [0.42970700836450487]
  We introduce a dataset-centric meta-evaluation framework that audits benchmark datasets at the sample level along five latent dimensions.<n>We show how these annotations enable criterion-driven orchestration of composite benchmark subsets across datasets.
  arXiv  Detail & Related papers  (2026-07-30T19:51:55Z)
• CEL: Comprehensive Counterfactual Explanations Library and Benchmark [0.0]
  We introduce CEL (Counterfactual Explanations Library), a unified library and benchmark for counterfactual explanations.<n>CEL includes 18 datasets of varying size and complexity and provides implementations or reimplementations of 14 widely used counterfactual methods.<n>CEL is the first comprehensive benchmark that systematically evaluates recent counterfactual explanation methods.
  arXiv  Detail & Related papers  (2026-07-24T07:21:27Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• DOCUEVAL: An LLM-based AI Engineering Tool for Building Customisable Document Evaluation Workflows [14.403156040714817]
  We present DOCUEVAL, an AI engineering tool for building customisable evaluators.<n>We show how DOCUEVAL enables both the engineering of evaluators and scalable, reliable document evaluation.
  arXiv  Detail & Related papers  (2025-09-12T08:09:09Z)
• AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume [2.2153783542347805]
  This paper investigates challenges in automatic text summarization evaluation.<n>Based on experiments conducted across six representative metrics, we highlight significant discrepancies between reported performances in the literature and those observed in our experimental setting.<n>We introduce a unified, open-source framework, applied to the SummEval dataset and designed to support fair and transparent comparison of evaluation metrics.
  arXiv  Detail & Related papers  (2025-08-29T08:05:00Z)
• Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments [0.9894420655516565]
  Document Retrieval-Augmented Fine-Tuning (DRAFT) is a novel approach that enhances the capabilities of a large language model (LLM) for safety-critical compliance assessment.<n>DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by introducing a novel fine-tuning framework.<n>Experiments with GPT-4o-mini demonstrate a 7% improvement in correctness over the baseline model.
  arXiv  Detail & Related papers  (2025-05-02T14:34:33Z)
• WritingBench: A Comprehensive Benchmark for Generative Writing [87.48445972563631]
  We present WritingBench, a benchmark designed to evaluate large language models (LLMs) across 6 core writing domains and 100, encompassing creative, persuasive, informative, and technical writing.<n>We propose a query-dependent evaluation framework that empowers LLMs to dynamically generate instance-specific assessment criteria.<n>This framework is complemented by a fine-tuned critic model for criteria-aware scoring, enabling evaluations in style, format and length.
  arXiv  Detail & Related papers  (2025-03-07T08:56:20Z)
• SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection [70.23196257213829]
  We propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection.<n>Our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains.<n>We then leverage large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels.
  arXiv  Detail & Related papers  (2025-03-05T09:37:05Z)
• StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs [78.84060166851805]
  StructTest is a novel benchmark that evaluates large language models (LLMs) on their ability to follow compositional instructions and generate structured outputs.<n> Assessments are conducted deterministically using a rule-based evaluator, which can be easily extended to new tasks and datasets.<n>We demonstrate that StructTest remains challenging even for top-performing models like Deepseek-V3/R1 and GPT-4o.
  arXiv  Detail & Related papers  (2024-12-23T22:08:40Z)
• OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain [62.89809156574998]
  We introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain.<n>Our benchmark is characterized by its multi-dimensional evaluation framework.<n>Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets.
  arXiv  Detail & Related papers  (2024-12-17T15:38:42Z)
• StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation [46.59416831869014]
  We propose a novel evaluation framework referred to as StructEval. Starting from an atomic test objective, StructEval deepens and broadens the evaluation by conducting a structured assessment across multiple cognitive levels and critical concepts. Experiments on three widely-used benchmarks demonstrate that StructEval serves as a reliable tool for resisting the risk of data contamination.
  arXiv  Detail & Related papers  (2024-08-06T16:28:30Z)
• CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists [15.19714327680248]
  We introduce CheckEval, a checklist-based evaluation framework that improves rating reliability via binary questions.<n>CheckEval dramatically improves the average agreement across evaluator models by 0.45 and reduces the score variance.
  arXiv  Detail & Related papers  (2024-03-27T17:20:39Z)
• Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization [13.736656652049884]
  Multimodal summarization aims to generate a concise summary based on the input text and image.<n>To evaluate the factuality of multimodal summarization models, we propose two fine-grained and explainable evaluation frameworks.
  arXiv  Detail & Related papers  (2024-02-18T01:03:25Z)
• fairlib: A Unified Framework for Assessing and Improving Classification Fairness [66.27822109651757]
  fairlib is an open-source framework for assessing and improving classification fairness. We implement 14 debiasing methods, including pre-processing, at-training-time, and post-processing approaches. The built-in metrics cover the most commonly used fairness criterion and can be further generalized and customized for fairness evaluation.
  arXiv  Detail & Related papers  (2022-05-04T03:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.