Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting

• URL: http://arxiv.org/abs/2606.09809v2
• Date: Tue, 09 Jun 2026 17:58:20 GMT
• Title: Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting
• Abstract summary: AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs.<n>Recent efforts address isolated components but leave three gaps: they cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record.<n>We present EvalCards, an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record.
• Score: 44.70980205167796
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. The cost is interpretive: readers cannot reliably compare results across sources, identify what a report omits, or trace an aggregate claim to its underlying evidence. Recent efforts address isolated components but leave three gaps: they cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record; they specify static representations that do not differentiate the questions different stakeholders bring to the same evidence; and they remain proposals on paper, lacking the extraction infrastructure required for adoption at scale. We present \EvalCards{}, an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record. We (1) derive a reporting schema from a structured review of 52 papers and 10 stakeholder interviews, (2) implement four interpretive signals (reproducibility, documentation completeness, provenance and risk, and score comparability), rendered through reader modes calibrated to research and non-research audiences, and (3) deploy a monitoring tool that applies \EvalCards{} across 5,816 models, 635 benchmarks, and 101,843 results, surfacing systematic gaps in current reporting practice.

Related papers

• TRACE: An Operational Reasoning Schema for Auditable Agentic Commitments [0.0]
  TRACE (Typed Reasoning And Commitment Evidence) is a typed, versioned schema for recording reasoning traces.<n>The paper argues in three layers that reasoning is not in the language model.<n>A record-consumer contract states what a record guarantees and what a consumer must honor in return.
  arXiv  Detail & Related papers  (2026-07-14T08:08:21Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• Rollout Cards: A Reproducibility Standard for Agent Research [15.381365113892848]
  Papers compare systems by reported scores while leaving rollout records behind those scores difficult to inspect.<n>For agentic tasks, this matters because the same behaviour can receive different reported scores when evaluations select different parts of a rollout or apply different reporting rules.<n>In a structured audit of 50 popular training and evaluation repositories, we find that none report how many runs failed, errored, or were skipped alongside headline scores.
  arXiv  Detail & Related papers  (2026-05-12T13:54:31Z)
• Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone [11.663456969895462]
  Alignment evaluation in machine learning has largely become evaluation of models.<n>This paper argues that deployment-relevant alignment cannot be inferred from model-level evaluation alone.
  arXiv  Detail & Related papers  (2026-05-06T03:28:30Z)
• FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification [57.196748998757954]
  We present FactReview, an evidence-grounded reviewing system that combines claim extraction, literature positioning, and execution-based claim verification.<n>Given a submission, FactReview identifies major claims and reported results, retrieves nearby work to clarify the paper's technical position, and, when code is available, executes the released repository.<n>It then produces a concise review and an evidence report that assigns each major claim one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive.
  arXiv  Detail & Related papers  (2026-04-05T11:45:22Z)
• HighlightBench: Benchmarking Markup-Driven Table Reasoning in Scientific Documents [88.16322879912458]
  multimodal large language models (MLLMs) have made substantial progress in document understanding, but their ability to treat such cues as explicit logical directives remains under-explored.<n>We introduce HighlightBench, a diagnostic benchmark for markup-driven table understanding that decomposes evaluation into five task families.<n>We show that even strong models remain unstable when visual cues must be consistently aligned with symbolic reasoning under structured output constraints.
  arXiv  Detail & Related papers  (2026-03-25T06:15:40Z)
• OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment [63.662126457336534]
  OpenNovelty is an agentic system for transparent, evidence-based novelty analysis.<n>It grounds all assessments in retrieved real papers, ensuring verifiable judgments.<n>OpenNovelty aims to empower the research community with a scalable tool that promotes fair, consistent, and evidence-backed peer review.
  arXiv  Detail & Related papers  (2026-01-04T15:48:51Z)
• Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It [1.6261897792391753]
  We conduct a systematic audit of three widely used reasoning benchmarks, SocialIQa, FauxPas-EAI, and ToMi.<n>We uncover pervasive flaws in both benchmark items and evaluation methodology.
  arXiv  Detail & Related papers  (2025-06-30T13:57:28Z)
• SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits [31.98028879922584]
  We introduce SummExecEdit, a novel pipeline and benchmark to assess models on their ability to both detect factual errors and provide accurate explanations.<n>The top-performing model, Claude3-Opus, achieves a joint detection and explanation score of only 0.49 in our benchmark.<n>We identify four primary types of explanation errors, with 45.4% of them involving a focus on completely unrelated parts of the summary.
  arXiv  Detail & Related papers  (2024-12-17T23:26:44Z)
• How well do you know your summarization datasets? [11.992125069326772]
  We analyze 600 samples from three popular summarization datasets. We follow with a thorough analysis of 27 state-of-the-art summarization models and 5 popular metrics.
  arXiv  Detail & Related papers  (2021-06-21T19:44:06Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)
• Unsupervised Opinion Summarization with Noising and Denoising [85.49169453434554]
  We create a synthetic dataset from a corpus of user reviews by sampling a review, pretending it is a summary, and generating noisy versions thereof. At test time, the model accepts genuine reviews and generates a summary containing salient opinions, treating those that do not reach consensus as noise.
  arXiv  Detail & Related papers  (2020-04-21T16:54:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.