The Evaluation Differential: When Frontier AI Models Recognise They Are Being Tested

• URL: http://arxiv.org/abs/2605.11496v1
• Date: Tue, 12 May 2026 04:13:35 GMT
• Title: The Evaluation Differential: When Frontier AI Models Recognise They Are Being Tested
• Abstract summary: Recent published evidence shows that contemporary AI models can recognise evaluation contexts, latently represent them, and behave differently under those contexts than under deployment-continuous conditions.<n>We argue that these findings create a claim-validity problem for safety conclusions drawn from frontier evaluations.<n>We develop a typology of safety claims (ED-stable, ED-degraded, ED-inverted, ED-undetermined) by their warrant-status under documented divergence, and specify TRACE (Test-Recognition Audit for Claim Evaluation).<n>We apply the framework retrospectively to three publicly documented evaluation incidents and discuss governance implications for system cards, conformity assessment, and
• Score: 11.663456969895462
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent published evidence from frontier laboratories shows that contemporary AI models can recognise evaluation contexts, latently represent them, and behave differently under those contexts than under deployment-continuous conditions. Anthropic's BrowseComp incident, the Natural Language Autoencoder findings on SWE-bench Verified and destructive-coding evaluations, and the OpenAI / Apollo anti-scheming work all document instances of this phenomenon. We argue that these findings create a claim-validity problem for safety conclusions drawn from frontier evaluations. We introduce the Evaluation Differential (ED), a conditional divergence in a target behavioural property between recognised-evaluation and deployment-continuous contexts, define a normalised effect-size form (nED) for cross-property comparison, and prove that marginal evaluation scores cannot identify ED. We develop a typology of safety claims (ED-stable, ED-degraded, ED-inverted, ED-undetermined) by their warrant-status under documented divergence, and specify TRACE (Test-Recognition Audit for Claim Evaluation), an audit protocol that wraps existing evaluation infrastructure and produces restricted claims rather than capability scores. We apply the framework retrospectively to three publicly documented evaluation incidents and discuss governance implications for system cards, conformity assessment, and the international network of AI safety and security institutes. TRACE does not eliminate adversarial adaptation; it disciplines the claims drawn from evaluation evidence by making explicit the conditions under which that evidence was produced.

Related papers

• Cognitive Admission Control: Risk-Conditioned Assurance for Consequential Actions in Agentic Distributed Systems [2.124730017640531]
  In agentic distributed systems, an agent may be authorized to mutate external infrastructure while lacking evidence that the mutation is ready to execute.<n> Cognitive Admission Control (CAC) makes this evidence requirement explicit.<n>A policy maps a typed action and its modeled risk to assurance obligations specifying predicates, evidence classes, scope, freshness, and witness-set constraints.<n>A successful admission produces a certificate binding the action, its witness manifest, and dispatch-time guards.
  arXiv  Detail & Related papers  (2026-09-14T20:20:54Z)
• When Compliance Data Masquerades as Evaluation: Measurement Validity for Deployed AI Systems [1.1957890510931166]
  We argue that a recurring failure in the evaluation of deployed AI systems occurs when data are interpreted as if they were designed for comparative evaluation.<n>We frame this issue as a measurement-validity problem in AI evaluation rather than as a transportation-specific data limitation.
  arXiv  Detail & Related papers  (2026-09-12T01:31:35Z)
• AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigation [48.55515767840701]
  "AI slop" is artifacts, hallucinated vulnerabilities, plausible but incorrect patches, and semantically repackaged bug reports.<n>This paper surveys the empirical evidence, identifies a unifying mechanism, and traces a path toward trustworthy triage.
  arXiv  Detail & Related papers  (2026-08-26T11:48:38Z)
• Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity [0.0]
  This paper introduces adversarial pragmatics as a benchmark for evaluating model behaviour under instruction conflict, embedded commands, quotation, scope ambiguity, deixis, indirect speech acts, and multi-turn agent transcripts.<n>The contribution is empirical and methodological: a linguistically controlled taxonomy, an 18-item seed benchmark with validator-enforced metadata, a 54-row local seed pilot, an expert-evaluation protocol distinguishing task success, policy compliance, safety risk, refusal outcome, and evaluator confidence, and metrics for judge validity, diagnostic ambiguity, and taxonomy drift.
  arXiv  Detail & Related papers  (2026-07-01T16:33:14Z)
• Position: Anthropomorphic Misalignment Research Needs Stronger Evidence [52.76826423649968]
  We show how conceptual ambiguity, non-robust datasets, experimental design, and insufficient causal interventions can lead to overinterpretation of model behaviors.<n>This position paper aims to offer guidance on evidentiary considerations that can help improve methodological rigor in AMR.
  arXiv  Detail & Related papers  (2026-05-29T16:38:53Z)
• Models That Know How Evaluations Are Designed Score Safer [38.21092181000792]
  We investigate evaluation meta-knowledge, defined as parametric knowledge about the structural traits that characterize evaluations.<n>Similar to contamination dataset, where benchmark exposure leads to higher performance through memorization, we hypothesize that models trained on texts describing evaluation practices may implicitly learn to recognize and respond to evaluation-like contexts.<n>Our results demonstrate that evaluation meta-knowledge may inflate safety benchmark performance, introducing a novel confounder that is independent of explicit memorization or verbalized evaluation awareness.
  arXiv  Detail & Related papers  (2026-05-27T15:11:35Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• Making AI-Assisted Grant Evaluation Auditable without Exposing the Model [0.0]
  We propose a TEE-based architecture that helps reconcile requirements through remote attestation.<n>The architecture allows an external verifier to check which model, rubric, prompt template, and input representation were used.<n>We position the design relative to confidential AI inference, attestable AI audits, zero-knowledge machine learning, algorithmic accountability, and AI-assisted peer review.
  arXiv  Detail & Related papers  (2026-04-28T04:10:04Z)
• Designing escalation criteria for international AI incident response: criteria, triggers, and thresholds [0.0]
  No operational criteria exist for determining when a detected AI incident warrants escalation beyond national handling to international coordination.<n>This paper proposes an escalation framework to address this gap, intended as a common reference point across jurisdictions.
  arXiv  Detail & Related papers  (2026-04-25T07:31:58Z)
• DREAM: Deep Research Evaluation with Agentic Metrics [21.555357444628044]
  We propose DREAM (Deep Research Evaluation with Agentic Metrics), a framework that makes evaluation itself agentic.<n> DREAM structures assessment through an evaluation protocol combining query-agnostic metrics with adaptive metrics generated by a tool-calling agent.<n>Controlled evaluations demonstrate DREAM is significantly more sensitive to factual and temporal decay than existing benchmarks.
  arXiv  Detail & Related papers  (2026-02-21T19:14:31Z)
• Alignment Verifiability in Large Language Models: Normative Indistinguishability under Behavioral Evaluation [0.0]
  We study alignment evaluation through the lens of statistical identifiability under partial observability.<n>We formalize the Alignment Verifiability Problem and introduce Normative Indistinguishability.<n>Our results show that behavioral benchmarks provide necessary but insufficient evidence for latent alignment under evaluation awareness.
  arXiv  Detail & Related papers  (2026-02-05T13:40:56Z)
• Towards Real-Time Fake News Detection under Evidence Scarcity [66.58597356379907]
  We propose Evaluation-Aware Selection of Experts (EASE), a novel framework for real-time fake news detection.<n>EASE adapts its decision-making process according to the assessed sufficiency of available evidence.<n>We introduce RealTimeNews-25, a new benchmark for evaluating model generalization on emerging news with limited evidence.
  arXiv  Detail & Related papers  (2025-10-13T11:11:46Z)
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
  We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
  arXiv  Detail & Related papers  (2025-08-29T08:48:00Z)
• Advancing Neural Network Verification through Hierarchical Safety Abstract Interpretation [52.626086874715284]
  We introduce a novel problem formulation called Abstract DNN-Verification, which verifies a hierarchical structure of unsafe outputs.<n>By leveraging abstract interpretation and reasoning about output reachable sets, our approach enables assessing multiple safety levels during the formal verification process.<n>Our contributions include a theoretical exploration of the relationship between our novel abstract safety formulation and existing approaches.
  arXiv  Detail & Related papers  (2025-05-08T13:29:46Z)
• SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection [70.23196257213829]
  We propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection.<n>Our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains.<n>We then leverage large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels.
  arXiv  Detail & Related papers  (2025-03-05T09:37:05Z)
• Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking [11.300523252168327]
  Evtextsuperscript2R combines the strengths of reference-based evaluation and verdict-level proxy scoring.<n>Evtextsuperscript2R consistently outperforms existing scoring approaches in accuracy and robustness.
  arXiv  Detail & Related papers  (2024-11-08T07:05:06Z)
• Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
  We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.