Detecting and Localizing Segment-Level Poisoning in Multi-Source LLM-Agent Inputs

• URL: http://arxiv.org/abs/2609.14723v1
• Date: Sun, 13 Sep 2026 18:31:11 GMT
• Title: Detecting and Localizing Segment-Level Poisoning in Multi-Source LLM-Agent Inputs
• Abstract summary: ActProbe is an internal-state-based framework for detecting and localizing poisoned segments in multi-source inputs.<n>It achieves a 0.01 false-positive rate, a 0.05 false-negative rate, and a 0.90 localization F1-score.
• Score: 10.050415083224532
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modern large language model (LLM) agents often construct prompts by aggregating retrieved passages, user reviews, and documents from multiple external sources. This paradigm exposes them to segment-level poisoning attacks, in which an adversary controlling only a small subset of sources injects malicious content to manipulate model outputs. Existing defenses mainly rely on textual patterns, external embeddings, or auxiliary detectors and may therefore fail against fluent, semantically plausible poisoned segments. They also provide limited support for locating the responsible segments. We observe that successful corrupted-evidence and adversarial-instruction attacks induce structured shifts in the LLM's internal activations, forming a consistent activation-space pattern that we call the poison direction. Based on this observation, we propose ActProbe, an internal-state-based framework for detecting and localizing poisoned segments in multi-source LLM inputs. ActProbe projects MLP activations onto a learned poison direction and uses a lightweight linear SVM trained on a small calibration set to detect contaminated prompts. It then applies BinRoL, which combines recursive replacement ablation, Mahalanobis-distance-based branch pruning, and MAD-based robust leaf detection to locate poisoned segments. ActProbe requires no modification to the backend LLM and reduces localization overhead from O(n) exhaustive probing to O(k log n) forward passes. Across three datasets, two attacks, and four open-weight LLMs, ActProbe achieves a 0.01 false-positive rate, a 0.05 false-negative rate, 0.94 localization recall, and a 0.90 localization F1-score. It remains effective against defense-aware adaptive attacks and can protect black-box APIs through surrogate-based poisoned-segment removal.

Related papers

• What Guides the Agent? Adjudicating Unauthorized Behavior via Localizing Behavior-Guiding Instructions [16.444770499696457]
  We propose Attnlocate, a runtime framework for fine-grained localization of context spans that genuinely influence tool-calling decisions.<n>Attnlocate casts this localization problem as an object detection task, aiming to detect the distinctive activation traces induced by behavior-guiding instructions.<n>We evaluate Attnlocate across ten agent configurations from five LLM families, covering scenarios involving indirect prompt injection and tool poisoning.
  arXiv  Detail & Related papers  (2026-08-25T03:24:53Z)
• CSO-LLM: Class Subspace Orthogonalization for Post-Training Backdoor Detection and Trigger Inversion in LLMs [7.4121421815806015]
  backdoor detection and trigger inversion schemes have been developed for AIs used e.g. for images.<n>We develop a highly effective detection and inversion framework for LLMs treated as classifiers.<n>One version of our detector performs continuous optimization in token embedding space, while a companion trigger-inversion and detection method performs greedy accretion in discrete token space.
  arXiv  Detail & Related papers  (2026-06-30T08:19:00Z)
• Curvature-Guided Module Localization for Low-Rank Detoxification of Backdoored Large Language Models [9.402151821242951]
  Backdoor attacks pose a serious threat to large language models (LLMs)<n>We propose a mechanistically guided weight-space repair framework that first localizes modules involved in propagating trigger-induced behavior.<n>Results show that the proposed approach substantially suppresses trigger-conditioned malicious responses while preserving benign model behavior.
  arXiv  Detail & Related papers  (2026-06-29T20:40:21Z)
• ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety [54.4092272526747]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success in cross-modal understanding and generation, yet their deployment is threatened by critical safety vulnerabilities.<n>We propose ProjLens, an interpretability framework designed to demystify MLLMs backdoors.
  arXiv  Detail & Related papers  (2026-04-21T04:52:38Z)
• AdversarialCoT: Single-Document Retrieval Poisoning for LLM Reasoning [115.3243260783674]
  Retrieval-augmented generation (RAG) enhances large language model (LLM) reasoning by retrieving external documents.<n>We study knowledge-base poisoning attacks in RAG, where an attacker injects malicious content into the retrieval corpus.<n>We propose AdversarialCoT, a query-specific attack that poisons only a single document in the corpus.
  arXiv  Detail & Related papers  (2026-04-14T02:10:23Z)
• Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs [2.2448294058653455]
  adversarial prompts exploit indirect input channels such as emails or user-generated content to circumvent alignment safeguards.<n>We propose Zero-Shot Embedding Drift Detection (ZEDD), a lightweight, low-engineering-overhead framework that identifies both direct and indirect prompt injection attempts.<n>ZEDD operates without requiring access to model internals, prior knowledge of attack types, or task-specific retraining.
  arXiv  Detail & Related papers  (2026-01-18T11:33:35Z)
• Lethe: Purifying Backdoored Large Language Models with Knowledge Dilution [49.78359632298156]
  Large language models (LLMs) have seen significant advancements, achieving superior performance in various Natural Language Processing (NLP) tasks.<n> backdoor attacks, where models behave normally for standard queries but generate harmful responses or unintended output when specific triggers are activated.<n>We present LETHE, a novel method to eliminate backdoor behaviors from LLMs through knowledge dilution.
  arXiv  Detail & Related papers  (2025-08-28T17:05:18Z)
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs [54.90315421117162]
  We propose a novel poisoning method via completely harmless data.<n>Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix.<n>We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning [76.82121084745785]
  Believe Your Eyes (BYE) is a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples.<n>It achieves near-zero attack success rates while maintaining clean-task performance.
  arXiv  Detail & Related papers  (2025-05-22T17:11:58Z)
• Defending against Indirect Prompt Injection by Instruction Detection [109.30156975159561]
  InstructDetector is a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks.<n>InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark.
  arXiv  Detail & Related papers  (2025-05-08T13:04:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.