Evaluation Blindness: How Silent Measurement Failures Corrupt AI Systems from Training to Deployment

• URL: http://arxiv.org/abs/2608.02786v1
• Date: Mon, 03 Aug 2026 18:33:37 GMT
• Title: Evaluation Blindness: How Silent Measurement Failures Corrupt AI Systems from Training to Deployment
• Abstract summary: The failure propagates through training loops, evaluation pipelines, and production monitoring stacks until downstream harm makes it visible.<n>This paper introduces evaluation blindness: a measurement function M exhibits evaluation blindness with respect to failure class F when it produces readings indistinguishable from a healthy state while the system is actually failing.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI systems can fail silently. The failure propagates through training loops, evaluation pipelines, and production monitoring stacks until downstream harm makes it visible. This paper introduces evaluation blindness: a measurement function M exhibits evaluation blindness with respect to failure class F when it produces readings indistinguishable from a healthy state while the system is actually failing, with no auxiliary signal flagging the gap. The problem surfaces at two lifecycle stages the literature has treated separately. At training time, reward models are gamed, importance-sampling corrections are silently miscalculated, and benchmark contamination inflates fine-tuning evaluations, all while loss curves look healthy and gradient updates proceed normally. At deployment time, monitoring fails to catch six classes of production failure, including an Operational category that is 100% silent by structural definition. We provide a formal detectability predicate unifying both stages. Four training-time case studies trace concrete breakdowns, including a real implementation bug in TRL PR #6594 where gradients are corrupted as loss decreases normally. A six-class taxonomy validated against 50 real-world incidents from court documents and regulatory filings finds that 53% of verifiable public failures were silent. A failure budget framework ties acceptable failure rates to use-case risk class. The implication is direct: measurement infrastructure is a correctness concern across the full AI lifecycle, not just at evaluation time. Data, code, and taxonomy schema are at https://github.com/priyanka25aug/llm-failure-taxonomy.

Related papers

• When Validation Stops Learning: Auditing Update Admission for Continual Embodied Agents [8.579680185421324]
  We argue that update admission must be assessed through both error control and retained learning opportunities.<n>A standard paired-binomial construction reduces this burden when outcome disagreements are rare.<n>A separate learned-dynamics stress test distinguishes model bias from feedback-selection error.
  arXiv  Detail & Related papers  (2026-09-09T22:25:16Z)
• DiagLoop: A Counterfactual Data Flywheel with Stage-Localized Reinforcement for Diagnostic LLMs [5.241146164724621]
  We present DiagLoop, a counterfactual data flywheel that converts codified physical relations or clinical guidelines into training supervision.<n>A training-only teacher proposes counterfactual worlds by varying causes, contexts, and observations, while an independent hybrid checker admits only valid worlds.<n>For nonterminal failures, a bounded repair probes downstream competence, and the resulting weakness profile guides subsequent data generation.
  arXiv  Detail & Related papers  (2026-08-04T13:49:41Z)
• Label-free Industrial Fault Detection via Adversarial Inverse Reinforcement Learning: A System for Run-to-Failure Prognostics [6.350348168407884]
  Machinery fault detection remains heavily reliant on supervised learning.<n>Current RL'' MFD methods reduce the problem to a static contextual bandit (CB) formulation.<n>We propose an adversarial inverse reinforcement learning framework that treats MFD as an offline IRL problem.
  arXiv  Detail & Related papers  (2026-07-25T02:02:22Z)
• DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation [50.584834483403675]
  Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone.<n>We demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface.<n>We introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them.
  arXiv  Detail & Related papers  (2026-06-30T20:57:51Z)
• Auditing Demonstration Curation Metrics: Action-Only Scorers Fail on the Structural Defects That Degrade Imitation Policies [0.0]
  Imitation-learning policies inherit the quality of the demonstrations they are trained on.<n>A growing set of curation metrics promise to score and filter low-quality demonstrations automatically.<n>We build a controlled testbed in which demonstration defects are injected with known type.
  arXiv  Detail & Related papers  (2026-06-04T02:07:27Z)
• Causal Label Recovery in Payment Networks [0.0]
  Fraud detection models in payment networks train on chargeback labels that are systematically biased.<n>A companion paper [arXiv05:26.27557] proved that these four impairments impose a minimax lower bound on detection performance.<n>We formalize the observation pipeline as a sequential missing-data problem with three stages and a corruption layer, and construct the Sequential Triply Robust estimator.
  arXiv  Detail & Related papers  (2026-05-28T02:43:34Z)
• Position: Early-Stage Quality Assurance in Annotation Pipelines Is More Cost-Effective Than Late-Stage Validation [17.77280120142941]
  This position paper argues that the machine learning community should prioritize early-stage quality assurance in annotation pipelines over the prevailing practice of late-stage validation.<n>We propose a taxonomy of three QA trigger points, namely pre-annotation (T0), post-annotation (T1), and post-review (T2), that decompose annotation into discrete validation opportunities.
  arXiv  Detail & Related papers  (2026-05-15T08:07:01Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• A Granular Study of Safety Pretraining under Model Abliteration [64.24346997570275]
  We study model abliteration, a lightweight projection technique designed to remove refusal-sensitive directions.<n>We issue 100 prompts with balanced harmful and harmless cases, classify responses as **Refusal** or **Non-Refusal** using multiple judges, and validate judge fidelity.<n>Our study produces a checkpoint-level characterization of which data-centric safety components remain robust under abliteration.
  arXiv  Detail & Related papers  (2025-10-03T07:01:45Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Continual Learning for Fake Audio Detection [62.54860236190694]
  This paper proposes detecting fake without forgetting, a continual-learning-based method, to make the model learn new spoofing attacks incrementally. Experiments are conducted on the ASVspoof 2019 dataset.
  arXiv  Detail & Related papers  (2021-04-15T07:57:05Z)
• Detecting Rewards Deterioration in Episodic Reinforcement Learning [63.49923393311052]
  In many RL applications, once training ends, it is vital to detect any deterioration in the agent performance as soon as possible. We consider an episodic framework, where the rewards within each episode are not independent, nor identically-distributed, nor Markov. We define the mean-shift in a way corresponding to deterioration of a temporal signal (such as the rewards), and derive a test for this problem with optimal statistical power.
  arXiv  Detail & Related papers  (2020-10-22T12:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.