When benchmark inferences do not compose: Projectibility in AI evaluation

• URL: http://arxiv.org/abs/2607.26159v1
• Date: Tue, 28 Jul 2026 18:07:04 GMT
• Title: When benchmark inferences do not compose: Projectibility in AI evaluation
• Abstract summary: An AI benchmark result rarely reaches a consequential claim in one step.<n>Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site.<n>This paper identifies a further problem: warranted links don't automatically make a warranted chain.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An AI benchmark result rarely reaches a consequential claim in one step. Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site, and combine it with assumptions about human review and downstream consequences. Validity-centred approaches require evidence for each claim. This paper identifies a further epistemic problem: warranted links don't automatically make a warranted chain. The target of one study may not be the source of the next; system, population, outcome, or conditions may change at the interface; and shared data or model lineage may make apparently independent support dependent. Projectibility concerns whether a bounded extension from observed to unobserved cases is warranted. Goodman supplies the problem of rival extensions; argument-based validity supplies an architecture for testing them. The paper's distinctive claim is a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align and dependence and uncertainty are carried through. A legal-research case shows how benchmark evidence and a deployment study can each be sound while remaining parallel. A reanalysis and simulation show why aggregate stability can erase distinctions a later projection requires. The resulting projectibility audit diagnoses unsupported joins in benchmark-to-use arguments.

Related papers

• Twin Worlds: Equivariance-Based Abstention for Evidence-Grounded Reasoning [38.63002496590365]
  We propose a framework for improving reliability in knowledge-intensive reasoning through equivariance-based abstention.<n>A key contributing factor is that entity mentions in context activate memorised associations, causing models to generate plausible responses ungrounded in evidence.<n>We construct multiple worlds via typed substitutions of the original input that preserve relational structure while reducing parametric priors.
  arXiv  Detail & Related papers  (2026-08-28T07:33:29Z)
• Structure-Preserving Uncertainty Propagation in First-Order Proof Search [0.2538209532048867]
  GK is a query-directed first-order prover that extends ordinary resolution-based proof search.<n>It works directly with non-ground clauses, including equality and function terms.<n>This paper adds structure-preserving quantitative reporting to that framework.
  arXiv  Detail & Related papers  (2026-08-10T07:00:55Z)
• Beyond Relevance: Bayesian Evidence Acquisition for Agentic Whole-Slide Image Reasoning [72.86819994769634]
  Whole-slide image (WSI) reasoning requires an agent to sequentially acquire visual evidence before answering a diagnostic question.<n>We propose BEACON, a plug-and-play agentic framework that reformulates WSI reasoning as a Bayesian evidence acquisition problem.
  arXiv  Detail & Related papers  (2026-08-06T08:40:57Z)
• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research [52.87128503345243]
  We introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research.<n>HiEviDR-Bench represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation.<n>We develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness.
  arXiv  Detail & Related papers  (2026-07-27T23:50:46Z)
• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• The Digital Twin Counterfactual Framework: A Validation Architecture for Simulated Potential Outcomes [0.0]
  Digital Twin Counterfactual Framework (DTCF)<n>Five-level validation architecture converts unfalsifiable claim that simulator produces correct counterfactuals into falsifiable tests against observable data.<n> decomposition separates causal quantities into those that are marginally validated.<n>Joint causal claims become explicitly assumption-indexed.
  arXiv  Detail & Related papers  (2026-04-01T19:04:15Z)
• Causality is Key for Interpretability Claims to Generalise [35.833847356014154]
  Interpretability research on large language models (LLMs) has yielded important insights into model behaviour.<n> recurring pitfalls persist: findings that do not generalise, and causal interpretations that outrun the evidence.<n>Pearl's causal hierarchy clarifies what an interpretability study can justify.
  arXiv  Detail & Related papers  (2026-02-18T18:45:04Z)
• CASA: Causality-driven Argument Sufficiency Assessment [79.13496878681309]
  We propose CASA, a zero-shot causality-driven argument sufficiency assessment framework. PS measures how likely introducing the premise event would lead to the conclusion when both the premise and conclusion events are absent. Experiments on two logical fallacy detection datasets demonstrate that CASA accurately identifies insufficient arguments.
  arXiv  Detail & Related papers  (2024-01-10T16:21:18Z)
• Nonparametric Identifiability of Causal Representations from Unknown Interventions [63.1354734978244]
  We study causal representation learning, the task of inferring latent causal variables and their causal relations from mixtures of the variables. Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data.
  arXiv  Detail & Related papers  (2023-06-01T10:51:58Z)
• Claim-Dissector: An Interpretable Fact-Checking System with Joint Re-ranking and Veracity Prediction [4.082750656756811]
  We present Claim-Dissector: a novel latent variable model for fact-checking and analysis. We disentangle the per-evidence relevance probability and its contribution to the final veracity probability in an interpretable way. Despite its interpretable nature, our system results competitive with state-of-the-art on the FEVER dataset.
  arXiv  Detail & Related papers  (2022-07-28T14:30:06Z)
• Nested Counterfactual Identification from Arbitrary Surrogate Experiments [95.48089725859298]
  We study the identification of nested counterfactuals from an arbitrary combination of observations and experiments. Specifically, we prove the counterfactual unnesting theorem (CUT), which allows one to map arbitrary nested counterfactuals to unnested ones.
  arXiv  Detail & Related papers  (2021-07-07T12:51:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.