BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encoder

• URL: http://arxiv.org/abs/2603.11664v2
• Date: Mon, 16 Mar 2026 08:21:21 GMT
• Title: BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encoder
• Abstract summary: BackdoorIDS is a zero-shot, inference-time backdoor samples detection method for pretrained vision encoders.<n>It consistently outperforms existing defenses across diverse attack types, datasets, and model families.<n>BackdoorIDS is compatible with a wide range of encoder architectures, including CNNs, ViTs, CLIP, and LLaVA-1.5.
• Score: 13.008131372229458
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised and multimodal vision encoders learn strong visual representations that are widely adopted in downstream vision tasks and large vision-language models (LVLMs). However, downstream users often rely on third-party pretrained encoders with uncertain provenance, exposing them to backdoor attacks. In this work, we propose BackdoorIDS, a simple yet effective zero-shot, inference-time backdoor samples detection method for pretrained vision encoders. BackdoorIDS is motivated by two observations: Attention Hijacking and Restoration. Under progressive input masking, a backdoored image initially concentrates attention on malicious trigger features. Once the masking ratio exceeds the trigger's robustness threshold, the trigger is deactivated, and attention rapidly shifts to benign content. This transition induces a pronounced change in the image embedding, whereas embeddings of clean images evolve more smoothly across masking progress. BackdoorIDS operationalizes this signal by extracting an embedding sequence along the masking trajectory and applying density-based clustering such as DBSCAN. An input is flagged as backdoored if its embedding sequence forms more than one cluster. Extensive experiments show that BackdoorIDS consistently outperforms existing defenses across diverse attack types, datasets, and model families. Notably, it is a plug-and-play approach that requires no retraining and operates fully zero-shot at inference time, making it compatible with a wide range of encoder architectures, including CNNs, ViTs, CLIP, and LLaVA-1.5.

Related papers

• EntropyScan: Towards Model-level Backdoor Detection in LVLMs via Visual Attention Entropy [66.59724477993339]
  Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across various tasks, yet they remain vulnerable to backdoor attacks.<n>We propose EntropyScan, a lightweight and trigger-agnostic method for model-level backdoor detection in LVLMs.<n>Our code will be publicly available soon.
  arXiv  Detail & Related papers  (2026-05-15T08:01:32Z)
• ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety [54.4092272526747]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success in cross-modal understanding and generation, yet their deployment is threatened by critical safety vulnerabilities.<n>We propose ProjLens, an interpretability framework designed to demystify MLLMs backdoors.
  arXiv  Detail & Related papers  (2026-04-21T04:52:38Z)
• Test-Time Attention Purification for Backdoored Large Vision Language Models [23.890959327899925]
  We provide a new mechanistic understanding of backdoor behaviors in large vision-language models (LVLMs)<n>We propose CleanSight, a training-free, plug-and-play defense that operates purely at test time.<n>CleanSight significantly outperforms existing pixel-based purification defenses across diverse datasets and backdoor attack types.
  arXiv  Detail & Related papers  (2026-03-13T13:45:06Z)
• On the Adversarial Robustness of Discrete Image Tokenizers [56.377796750281796]
  We first formulate attacks that aim to perturb the features extracted by discrete tokenizers, and thus change the extracted tokens.<n>We fine-tune popular tokenizers with unsupervised adversarial training, keeping all other components frozen.<n>Our approach significantly improves robustness to both unsupervised and end-to-end supervised attacks and generalizes well to unseen tasks and data.
  arXiv  Detail & Related papers  (2026-02-20T14:39:17Z)
• BackdoorVLM: A Benchmark for Backdoor Attacks on Vision-Language Models [63.5775877701015]
  We introduce textbfBackdoorVLM, the first comprehensive benchmark for evaluating backdoor attacks on vision-language models (VLMs)<n>BackdoorVLM organizes multimodal backdoor threats into 5 representative categories: targeted refusal, malicious injection, jailbreak, concept substitution, and perceptual hijack.<n>We evaluate these threats using 12 representative attack methods spanning text, image, and bimodal triggers, tested on 2 open-source VLMs and 3 multimodal datasets.
  arXiv  Detail & Related papers  (2025-11-24T09:30:38Z)
• Backdoor Attacks on Open Vocabulary Object Detectors via Multi-Modal Prompt Tuning [5.0734761482919115]
  Open-vocabulary object detectors (OVODs) unify vision and language to detect arbitrary object categories based on text prompts.<n>We conduct the first study of backdoor attacks on OVODs and reveal a new attack surface introduced by prompt tuning.
  arXiv  Detail & Related papers  (2025-11-16T19:05:31Z)
• Proactive Disentangled Modeling of Trigger-Object Pairings for Backdoor Defense [0.0]
  Deep neural networks (DNNs) and generative AI (GenAI) are increasingly vulnerable to backdoor attacks.<n>In this paper, we introduce DBOM, a proactive framework that leverages structured disentanglement to identify and neutralize both seen and unseen backdoor threats.<n>We show that DBOM robustly detects poisoned images prior to downstream training, significantly enhancing the security of training pipelines.
  arXiv  Detail & Related papers  (2025-08-03T21:58:15Z)
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning [76.82121084745785]
  Believe Your Eyes (BYE) is a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples.<n>It achieves near-zero attack success rates while maintaining clean-task performance.
  arXiv  Detail & Related papers  (2025-05-22T17:11:58Z)
• Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models [70.03122709795122]
  Backdoor attacks targeting text-to-image diffusion models have advanced rapidly.<n>Current backdoor samples often exhibit two key abnormalities compared to benign samples.<n>We propose Trigger without Trace (TwT) by explicitly mitigating these consistencies.
  arXiv  Detail & Related papers  (2025-03-22T10:41:46Z)
• DeDe: Detecting Backdoor Samples for SSL Encoders via Decoders [6.698677477097004]
  Self-supervised learning (SSL) is pervasively exploited in training high-quality upstream encoders with a large amount of unlabeled data.<n>Victim encoders associate triggered inputs with target embeddings, such that the downstream tasks inherit unintended behaviors when the trigger is activated.<n>We propose a novel detection mechanism, DeDe, which detects the activation of backdoor mappings caused by triggered inputs on victim encoders.
  arXiv  Detail & Related papers  (2024-11-25T07:26:22Z)
• GhostEncoder: Stealthy Backdoor Attacks with Dynamic Triggers to Pre-trained Encoders in Self-supervised Learning [15.314217530697928]
  Self-supervised learning (SSL) pertains to training pre-trained image encoders utilizing a substantial quantity of unlabeled images. We propose GhostEncoder, the first dynamic invisible backdoor attack on SSL.
  arXiv  Detail & Related papers  (2023-10-01T09:39:27Z)
• CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning [63.72975421109622]
  CleanCLIP is a finetuning framework that weakens the learned spurious associations introduced by backdoor attacks. CleanCLIP maintains model performance on benign examples while erasing a range of backdoor attacks on multimodal contrastive learning.
  arXiv  Detail & Related papers  (2023-03-06T17:48:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.