Hallucination as Exploit: Evidence-Carrying Multimodal Agents

• URL: http://arxiv.org/abs/2605.19192v2
• Date: Wed, 20 May 2026 21:49:08 GMT
• Title: Hallucination as Exploit: Evidence-Carrying Multimodal Agents
• Abstract summary: Multimodal agents increasingly choose tool calls from screenshots, documents, and webpages, where a false perceptual claim can turn from an answer-quality error into an authorization failure.<n>We propose evidence-carrying multimodal agents (ECA), which treat free-form model text as inadmissible evidence.<n>ECA converts opaque model belief into auditable residuals at the verifier, schema, and implementation levels.
• Score: 10.441697487723568
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal agents increasingly choose tool calls from screenshots, documents, and webpages, where a false perceptual claim can turn hallucination from an answer-quality error into an authorization failure. We formalize this failure mode as hallucination-to-action conversion: an unsupported claim supplies the precondition for a privileged action. We propose evidence-carrying multimodal agents (ECA), which treat free-form model text as inadmissible evidence, decompose each tool call into action-critical predicates, obtain typed certificates from constrained DOM/OCR/AX verifiers, and use a deterministic gate to authorize only the privileges those certificates support. Rather than hiding perception error, ECA converts opaque model belief into auditable residuals at the verifier, schema, and implementation levels. Verifier red-teaming across 17 canonical attack categories shows that four targeted hardening steps are each necessary; after hardening, canonical gate bypass is 0/1,700 (Wilson 95% upper bound 0.22%). With content-derived certificates, ECA observes zero unsafe executions on 200 end-to-end tasks (Wilson 95% upper bound 2.67%) and 120 browser tasks (upper bound 4.3%). A HACR audit on 500 stratified task keys shows that unsupported action-critical claims reach unsafe execution for naive agents (100.0%) and prompt-only defenses (49.6%), but not for ECA. Oracle-certificate replay over 7,488 GPT-5.4 traces isolates gate correctness, while neural judge baselines still admit most unsafe actions under the same threat model. The resulting principle is simple: model language may propose tool use, but certified predicates must authorize it.

Related papers

• AcquireBound: Runtime Authorization for Resources Acquired by AI Agents [0.0]
  Post-fulfillment activation gap spans tool-mediated creation, inter-agent delegation, and agentic commerce.<n>We present AcquireBound, a provenance-bounded runtime authorization architecture.<n>We prove eight safety properties covering quarantine, backing, non-amplification, split non-evasion, crash/retry, refunds, epochs, and effect confinement.
  arXiv  Detail & Related papers  (2026-09-13T19:16:44Z)
• Recognition Without Enforcement: Configuration-Dependent Failures in LLM Agent Instruction Arbitration and External Control [0.0]
  LLM agents arbitrate among instructions from system prompts, users, memory, and tools, but this arbitration cannot be assumed to enforce trust boundaries.<n>We identify a recognition-enforcement gap: source-format features are linearly decodable from model activations, and models can explicitly identify forged authority when prompted.<n>We treat model self-arbitration as a capability rather than a security boundary and implement an external reference monitor combining authenticated source routing with capability-gated tool execution.
  arXiv  Detail & Related papers  (2026-08-28T16:34:05Z)
• Task-Conditioned Least-Privilege Learning for Executable Terminal and MCP Agents [45.88028371034407]
  Tool-using large language-model agents can complete a task while exercising authority that the user did not grant or the task does not need.<n>We propose a framework where each action is audited before execution and again from observed effects along six dimensions of risk.<n>We find that after training using this framework on Qwen3.5-4B over 1,500 tasks, the selected seed reaches 98.48% safe success across 2,896 evaluation episodes.
  arXiv  Detail & Related papers  (2026-08-18T22:07:23Z)
• Auditable Release Control for Pedagogical Leakage in LLM Tutors [0.0]
  Large language model tutors can be correct and helpful yet disclose an answer or decisive reasoning before that disclosure is authorized.<n>We formalize this state- and action-dependent failure as pedagogical leakage-aware authorization and introduce complete-mediation boundary.<n>A global A1 yields 0 majority and 54 any-utility flags, outperforming fitted Q on automatic safety and utility.
  arXiv  Detail & Related papers  (2026-08-01T08:24:30Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens [65.53494487819053]
  Claw-like AI agents (e.g., OpenClaw) are always-on processes with persistent access to credentials, files, tools, and external services.<n>We adopt a computer-system analogy: treating a Claw-like agent as an agentic computer system whose gateway runtime plays an OS-like mediation role.<n>We develop SafeClawArena, a benchmark of 406 adversarial tasks across four attack surfaces.
  arXiv  Detail & Related papers  (2026-06-29T18:00:45Z)
• Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks [40.270213696031625]
  OverEager-Gen is a benchmark dedicated to overeager behavior on benign tasks.<n>On Claude Code, stripping the consent declaration alone raises the overeager rate from 0.0% to 17.1%.<n>OverEager-Gen therefore certifies each scenario's discriminative power before admission.
  arXiv  Detail & Related papers  (2026-05-18T16:00:41Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning [0.9236074230806578]
  Oracle Poisoning is an attack class in which an adversary corrupts a structured knowledge graph.<n>Unlike prompt injection, Oracle Poisoning manipulates the data agents reason over, not their instructions.<n>We demonstrate six attack scenarios against a production 42-million-node code knowledge graph.
  arXiv  Detail & Related papers  (2026-05-10T23:55:13Z)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents [2.1942030377331245]
  Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets.<n>We introduce MOSAIC-Bench, a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates.<n>We show that nine production coding agents compose innocuous tickets at 53-86% end-to-end ASR with only two refusals across all staged runs.
  arXiv  Detail & Related papers  (2026-05-05T16:38:23Z)
• Certified Purity for Cognitive Workflow Executors: From Static Analysis to Cryptographic Attestation [0.0]
  A prior three-layer governance architecture proves governance completeness, provenance completeness, and the impossibility of ungoverned effects.<n>This paper closes the gap through four mechanisms.<n>We prove four theorems: structural purity by construction, bypass elimination for all five BEAM bypass classes, certificate integrity, and gate completeness.
  arXiv  Detail & Related papers  (2026-05-01T19:04:37Z)
• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• Governing Dynamic Capabilities: Cryptographic Binding and Reproducibility Verification for AI Agent Tool Use [0.0]
  No existing security layer verifies what an AI agent can do, whether it executed what it claims, or what happened in a multi-agent interaction.<n>Existing frameworks conflate the two, enabling silent capability escalation and leaving interactions without verifiable provenance.<n>We derive three Agent Governance Requirements: capability integrity (G1), behavioral verifiability (G2), and interaction auditability (G3)<n>We validate with two crypto-agnostic instantiations -- basic (Ed25519, SHA-256; 97 us verify) and enhanced (BBS+ selective disclosure, Groth16 DV-SNARK; 13.8 ms)
  arXiv  Detail & Related papers  (2026-03-15T11:46:57Z)
• Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents [0.0]
  NabaOS is a lightweight verification framework for interactive agents.<n>It classifies every claim in an AI response by its source.<n>It detects 94.2% of fabricated tool executions, 87.6% of misstating counts, and 78.4% of false absence claims.
  arXiv  Detail & Related papers  (2026-03-09T20:45:41Z)
• Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Models [71.44858461725893]
  Given a model fine-tuned by an untrusted third party, determining whether the model has been injected with a backdoor is a critical and challenging problem.<n>Existing detection methods usually rely on prior knowledge of training dataset, backdoor triggers and targets.<n>We introduce Assimilation Matters in DETection (AMDET), a novel model-level detection framework that operates without any such prior knowledge.
  arXiv  Detail & Related papers  (2025-11-29T06:20:00Z)
• Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Models [75.29749026964154]
  Ourmethod reduces the average Attack Success Rate to 4.41% across multiple benchmarks.<n>Clean accuracy and utility are preserved within 0.5% of the original model.<n>The defense generalizes across different types of backdoors, confirming its robustness in practical deployment scenarios.
  arXiv  Detail & Related papers  (2025-10-11T15:47:35Z)
• Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain [82.98626829232899]
  Fine-tuning AI agents on data from their own interactions introduces a critical security vulnerability within the AI supply chain.<n>We show that adversaries can easily poison the data collection pipeline to embed hard-to-detect backdoors.
  arXiv  Detail & Related papers  (2025-10-03T12:47:21Z)
• VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection [55.957275374847484]
  VulAgent is a multi-agent vulnerability detection framework based on hypothesis validation.<n>It implements a semantics-sensitive, multi-view detection pipeline, each aligned to a specific analysis perspective.<n>On average, VulAgent improves overall accuracy by 6.6%, increases the correct identification rate of vulnerable--fixed code pairs by up to 450%, and reduces the false positive rate by about 36%.
  arXiv  Detail & Related papers  (2025-09-15T02:25:38Z)
• Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable [70.77600345240867]
  A novel arbitrary-in-arbitrary-out (AIAO) strategy makes watermarks resilient to fine-tuning-based removal. Unlike the existing methods of designing a backdoor for the input/output space of diffusion models, in our method, we propose to embed the backdoor into the feature space of sampled subpaths. Our empirical studies on the MS-COCO, AFHQ, LSUN, CUB-200, and DreamBooth datasets confirm the robustness of AIAO.
  arXiv  Detail & Related papers  (2024-05-01T12:03:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.