Inverting Trojans in LLMs

• URL: http://arxiv.org/abs/2509.16203v1
• Date: Fri, 19 Sep 2025 17:59:57 GMT
• Title: Inverting Trojans in LLMs
• Abstract summary: backdoor detection and inversion schemes have been developed for AIs used e.g. for images.<n>We propose a trigger inversion approach with three key components.<n>Unlike many recent works, we demonstrate that our approach reliably detects and successfully inverts ground-truth backdoor trigger phrases.
• Score: 10.235298805782422
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While effective backdoor detection and inversion schemes have been developed for AIs used e.g. for images, there are challenges in "porting" these methods to LLMs. First, the LLM input space is discrete, which precludes gradient-based search over this space, central to many backdoor inversion methods. Second, there are ~30,000^k k-tuples to consider, k the token-length of a putative trigger. Third, for LLMs there is the need to blacklist tokens that have strong marginal associations with the putative target response (class) of an attack, as such tokens give false detection signals. However, good blacklists may not exist for some domains. We propose a LLM trigger inversion approach with three key components: i) discrete search, with putative triggers greedily accreted, starting from a select list of singletons; ii) implicit blacklisting, achieved by evaluating the average cosine similarity, in activation space, between a candidate trigger and a small clean set of samples from the putative target class; iii) detection when a candidate trigger elicits high misclassifications, and with unusually high decision confidence. Unlike many recent works, we demonstrate that our approach reliably detects and successfully inverts ground-truth backdoor trigger phrases.

Related papers

• CSO-LLM: Class Subspace Orthogonalization for Post-Training Backdoor Detection and Trigger Inversion in LLMs [7.4121421815806015]
  backdoor detection and trigger inversion schemes have been developed for AIs used e.g. for images.<n>We develop a highly effective detection and inversion framework for LLMs treated as classifiers.<n>One version of our detector performs continuous optimization in token embedding space, while a companion trigger-inversion and detection method performs greedy accretion in discrete token space.
  arXiv  Detail & Related papers  (2026-06-30T08:19:00Z)
• Caught in the Act(ivation): Toward Pre-Output and Multi-Turn Detection of Credential Exfiltration by LLM Agents [0.0]
  We study whether activation probes can detect credential access before output tokens are emitted.<n>Third, we treat multi-turn exfiltration as a cumulative information-flow problem.<n>Results suggest credential-exfiltration defenses should combine pre-output monitoring, canary detection, and temporal leakage accounting.
  arXiv  Detail & Related papers  (2026-06-02T18:53:17Z)
• ShadowLogic: Backdoors in Any Whitebox LLM [0.0]
  Large language models (LLMs) are widely deployed across various applications, often with safeguards to prevent the generation of harmful or restricted content.<n>This work highlights a critical security vulnerability in computational graph-based LLM formats.<n>We introduce ShadowLogic, a method for creating a backdoor in a white-box LLM by injecting an uncensoring vector into its computational graph representation.
  arXiv  Detail & Related papers  (2025-11-01T19:10:08Z)
• Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models [17.839413035304748]
  Backdoor unalignment attacks against Large Language Models (LLMs) enable the stealthy compromise of safety alignment using a hidden trigger.<n>We introduce BEAT, a black-box defense that detects triggered samples during inference to deactivate the backdoored LLM.<n>Our method addresses the challenges of sample-dependent targets from an opposite perspective.
  arXiv  Detail & Related papers  (2025-06-19T16:30:56Z)
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs [54.90315421117162]
  We propose a novel poisoning method via completely harmless data.<n>Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix.<n>We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors [77.82885394684202]
  We propose textbfContrastive textbfParaphrase textbfAttack (CoPA), a training-free method that effectively deceives text detectors.<n>CoPA constructs an auxiliary machine-like word distribution as a contrast to the human-like distribution generated by large language models.<n>Our theoretical analysis suggests the superiority of the proposed attack.
  arXiv  Detail & Related papers  (2025-05-21T10:08:39Z)
• CALM: Curiosity-Driven Auditing for Large Language Models [27.302357350862085]
  We propose Curiosity-Driven Auditing for Large Language Models (CALM) to finetune an LLM as the auditor agent.<n>CALM successfully identifies derogatory completions involving celebrities and uncovers inputs that elicit specific names under the black-box setting.
  arXiv  Detail & Related papers  (2025-01-06T13:14:34Z)
• Forking Paths in Neural Text Generation [14.75166317633176]
  We develop a novel approach to representing uncertainty dynamics across individual tokens of text generation.<n>We use our method to analyze LLM responses on 7 different tasks across 4 domains.<n>We find many examples of forking tokens, including surprising ones such as punctuation marks.
  arXiv  Detail & Related papers  (2024-12-10T22:57:57Z)
• Palisade -- Prompt Injection Detection Framework [0.9620910657090188]
  Large Language Models are vulnerable to malicious prompt injection attacks. This paper proposes a novel NLP based approach for prompt injection detection. It emphasizes accuracy and optimization through a layered input screening process.
  arXiv  Detail & Related papers  (2024-10-28T15:47:03Z)
• AdvQDet: Detecting Query-Based Adversarial Attacks with Adversarial Contrastive Prompt Tuning [93.77763753231338]
  Adversarial Contrastive Prompt Tuning (ACPT) is proposed to fine-tune the CLIP image encoder to extract similar embeddings for any two intermediate adversarial queries. We show that ACPT can detect 7 state-of-the-art query-based attacks with $>99%$ detection rate within 5 shots. We also show that ACPT is robust to 3 types of adaptive attacks.
  arXiv  Detail & Related papers  (2024-08-04T09:53:50Z)
• Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models [79.76293901420146]
  Large Language Models (LLMs) are employed across various high-stakes domains, where the reliability of their outputs is crucial. Our research investigates the fragility of uncertainty estimation and explores potential attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which, when activated by a specific trigger in the input, manipulates the model's uncertainty without affecting the final output.
  arXiv  Detail & Related papers  (2024-07-15T23:41:11Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes [61.916827858666906]
  Large Language Models (LLMs) are becoming a prominent generative AI tool, where the user enters a query and the LLM generates an answer. To reduce harm and misuse, efforts have been made to align these LLMs to human values using advanced training techniques such as Reinforcement Learning from Human Feedback. Recent studies have highlighted the vulnerability of LLMs to adversarial jailbreak attempts aiming at subverting the embedded safety guardrails. This paper proposes a method called Gradient Cuff to detect jailbreak attempts.
  arXiv  Detail & Related papers  (2024-03-01T03:29:54Z)
• TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification [41.25887364156612]
  We describe the novel fingerprinting problem of Black-box Identity Verification (BBIV) The goal is to determine whether a third-party application uses a certain LLM through its chat function. We propose a method called Targeted Random Adversarial Prompt (TRAP) that identifies the specific LLM in use.
  arXiv  Detail & Related papers  (2024-02-20T13:20:39Z)
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks [99.23352758320945]
  We propose SmoothLLM, the first algorithm designed to mitigate jailbreaking attacks on large language models (LLMs) Based on our finding that adversarially-generated prompts are brittle to character-level changes, our defense first randomly perturbs multiple copies of a given input prompt, and then aggregates the corresponding predictions to detect adversarial inputs.
  arXiv  Detail & Related papers  (2023-10-05T17:01:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.