Labels Are Not Endpoints: Treatment Leakage and Construct Validity in MCP Agent Security Evaluation

• URL: http://arxiv.org/abs/2608.12880v1
• Date: Thu, 13 Aug 2026 06:44:40 GMT
• Title: Labels Are Not Endpoints: Treatment Leakage and Construct Validity in MCP Agent Security Evaluation
• Abstract summary: We audit a preserved campaign by tracing 10,200 execution rows to 180 model-bound requests, 45 semantic requests, and 15 observable stimuli.<n>We contribute a seven-link Integrity Chain and an executable, scope-bounded endpoint-integrity linter.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Security evaluations of tool-using agents often equate stored labels with behavioral facts. We audit a preserved campaign by tracing 10,200 execution rows to 180 model-bound requests, 45 semantic requests, and 15 observable stimuli. Two schema treatments were delivered, but the planned external payload-family corpus was not. The historical grader exhibited direct treatment leakage: treatment metadata gated the ATTACK_SUCCESS class, so fixed behavior could change class under treatment relabeling. A treatment-blind reconstruction corrects 58 historical ATTACK_SUCCESS or HIJACK_ATTEMPT labels to authorized benign completions while preserving three verified protected-data transfers and one separate unauthorized-forwarding case. The locked v2 census contains exactly zero ATTACK_SUCCESS records, while the forwarding case remains a HIJACK_ATTEMPT at a semantic boundary concerning objective completion. A dual-reviewer blinded concordance review of all 96 requests deemed structurally interpretable by locked v2 produced identical reviewer-consensus classes but differed from the locked codebook on four construct-boundary cases. We contribute a seven-link Integrity Chain and an executable, scope-bounded endpoint-integrity linter. The result is a campaign-bounded measurement audit, not a population attack-rate, model-ranking, defense-efficacy, or causal estimate.

Related papers

• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• Calibrating Criterion Revision in LLM Agents: Failure Modes and a Trace-Anchored Protocol [3.202978695204522]
  Language-model agents can improve after failure or carry text across episodes without revising what counts as success.<n>We evaluate CMB-0.1 on twelve cross-domain cases and four arms.<n>No model trial satisfies all five conditions, but this zero does not establish general capability absence.
  arXiv  Detail & Related papers  (2026-08-21T04:21:28Z)
• Cross-View Correspondence Is a Measurement Intervention: Two-Sided Validation for Agent Evaluation and Credit Assignment [3.2479665905442645]
  Cross-view correspondence must be declared, validated, and propagated into uncertainty before agent evaluation or credit assignment supports a point conclusion.<n>We develop a validity theory and audit with three components: two-sided validation of nuisance removal and response preservation, all-optima identification of downstream conclusions, and uncertainty propagation after validity is established.
  arXiv  Detail & Related papers  (2026-08-18T12:39:21Z)
• Mind the Hook: Source-Level Auditing of Privacy Defenses in Retrieval-Augmented Generation [0.30586855806896046]
  Black-box privacy scores for retrieval-augmented generation (RAG) are difficult to interpret unless the defense's active pipeline hook is known.<n>We propose an active-path audit: inventory source-level hooks over retrieval, retrieved content, and generation; map each metric to the leakage channel it observes; and validate generated-text effects with exact-match canaries.
  arXiv  Detail & Related papers  (2026-08-10T01:40:25Z)
• What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents [0.0]
  guardrails act before irreversible tool calls, but their guarantees depend on what policy state is representable.<n>Pearson representation attacks add a margin, so benign calibration alone does not transfer.<n>Experiments target these distinctions through static diagnostics, controlled-model enumeration, representation rewrites, and paired closed-loop reruns.
  arXiv  Detail & Related papers  (2026-07-24T19:14:26Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Agent-Safety Evaluations as Load-Bearing Evidence: A Vendor-Neutral, Cross-Harness Reconstructability Metric [0.0]
  No vendor-neutral, runnable instrument scores reconstructability as an evaluation-validity metric.<n>This paper introduces a property-level reconstructability metric over eight decision-property classes.
  arXiv  Detail & Related papers  (2026-07-14T07:54:37Z)
• Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents [0.0]
  Agentic red-teaming benchmarks report whether an injected agent was compromised as a single bit: the attack succeeded, or it did not.<n>We argue that this binary attack-success rate discards the information a defender most needs, namely how harmful the resulting action was.<n>We introduce an action-graded harm rubric that scores an agent's tool-call trajectory on a seven-level ordinal scale (L0 to L6) according to whether the executed action was reversible, whether it crossed scope to reach another party, and whether it expanded privilege.
  arXiv  Detail & Related papers  (2026-07-08T14:38:01Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• The Misattribution Gap: When Memory Poisoning Looks Like Model Failure in Agentic AI Systems [1.0262304700896199]
  We formalize emphSemantic Norm Drift (SND) as a third path to agent misconduct.<n>In SND, a policy-formatted document enters a shared vector store through normal uploads and later reappears as trusted system context.<n>Counterfactual Composition Testing identifies the causal entry with 87.5% accuracy and zero false positives.
  arXiv  Detail & Related papers  (2026-05-12T20:21:47Z)
• TwinGate: Stateful Defense against Decompositional Jailbreaks in Untraceable Traffic via Asymmetric Contrastive Learning [60.68349524623048]
  Decompositional jailbreaks pose a critical threat to large language models.<n>We present TwinGate, a stateful dual-encoder defense framework.<n>We construct a comprehensive dataset of over 3.62 million instructions spanning 8,600 distinct malicious intents.
  arXiv  Detail & Related papers  (2026-04-30T13:44:01Z)
• INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection [40.157413213893165]
  Administrative phone tasks drain roughly 1 trillion USD annually from U.S. healthcare.<n>We introduce INSURE-Dial, the first public benchmark for developing and assessing compliance-aware voice agents.
  arXiv  Detail & Related papers  (2026-01-28T17:27:49Z)
• Detecting Adversarial Fine-tuning with Auditing Agents [38.964973163076586]
  We introduce the concept of a fine-tuning auditing agent and show it can detect harmful fine-tuning prior to model deployment.<n>We evaluate our detection approach on a diverse set of eight strong fine-tuning attacks from the literature, along with five benign fine-tuned models.<n>Most promising, the auditor is able to detect covert cipher attacks that evade safety evaluations and content moderation of the dataset.
  arXiv  Detail & Related papers  (2025-10-17T23:01:16Z)
• A Granular Study of Safety Pretraining under Model Abliteration [64.24346997570275]
  We study model abliteration, a lightweight projection technique designed to remove refusal-sensitive directions.<n>We issue 100 prompts with balanced harmful and harmless cases, classify responses as **Refusal** or **Non-Refusal** using multiple judges, and validate judge fidelity.<n>Our study produces a checkpoint-level characterization of which data-centric safety components remain robust under abliteration.
  arXiv  Detail & Related papers  (2025-10-03T07:01:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.