Backdoor Attacks on Prompt-Driven Video Segmentation Foundation Models

• URL: http://arxiv.org/abs/2512.22046v1
• Date: Fri, 26 Dec 2025 14:48:58 GMT
• Title: Backdoor Attacks on Prompt-Driven Video Segmentation Foundation Models
• Abstract summary: We propose BadVSFM, the first backdoor framework tailored to prompt-driven VSFMs.<n>BadVSFM uses a two-stage strategy: (1) steer the image encoder so triggered frames map to a designated target embedding while clean frames remain aligned with a clean reference encoder.<n>Experiments show that BadVSFM achieves strong, controllable backdoor effects under diverse triggers and prompts.
• Score: 21.15174669411218
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt-driven Video Segmentation Foundation Models (VSFMs) such as SAM2 are increasingly deployed in applications like autonomous driving and digital pathology, raising concerns about backdoor threats. Surprisingly, we find that directly transferring classic backdoor attacks (e.g., BadNet) to VSFMs is almost ineffective, with ASR below 5\%. To understand this, we study encoder gradients and attention maps and observe that conventional training keeps gradients for clean and triggered samples largely aligned, while attention still focuses on the true object, preventing the encoder from learning a distinct trigger-related representation. To address this challenge, we propose BadVSFM, the first backdoor framework tailored to prompt-driven VSFMs. BadVSFM uses a two-stage strategy: (1) steer the image encoder so triggered frames map to a designated target embedding while clean frames remain aligned with a clean reference encoder; (2) train the mask decoder so that, across prompt types, triggered frame-prompt pairs produce a shared target mask, while clean outputs stay close to a reference decoder. Extensive experiments on two datasets and five VSFMs show that BadVSFM achieves strong, controllable backdoor effects under diverse triggers and prompts while preserving clean segmentation quality. Ablations over losses, stages, targets, trigger settings, and poisoning rates demonstrate robustness to reasonable hyperparameter changes and confirm the necessity of the two-stage design. Finally, gradient-conflict analysis and attention visualizations show that BadVSFM separates triggered and clean representations and shifts attention to trigger regions, while four representative defenses remain largely ineffective, revealing an underexplored vulnerability in current VSFMs.

Related papers

• S$^3$AM: A Single-Stream SAM with Reliability-Calibrated Frequency Adapter for Multi-modal Salient Object Detection [74.15663758681849]
  Single-stream framework integrates reliability-calibrated frequency adaptation into the adopted SAM backbone for MSOD.<n>Experiments on RGB-D, RGB-T, and RGB-NIR salient object detection benchmarks validate that the proposed framework achieves competitive performance.
  arXiv  Detail & Related papers  (2026-08-18T08:05:00Z)
• SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models [24.580427340782524]
  Recent diffusion models (VDMs) synthesize visually convincing clips, yet still drop entities, mis-bind attributes, weaken the interactions specified in the prompt-alignment rubric.<n>We present SARA, Adaptive Representationally Alignment, which keeps token-relation on a frozen VFM target and adds a text-conditioned saliency that decides which token pairs carry supervision.
  arXiv  Detail & Related papers  (2026-05-08T14:36:32Z)
• Blind Bitstream-corrupted Video Recovery via Metadata-guided Diffusion Model [21.33395654187404]
  We introduce a new blind video recovery setting that removes the reliance on predefined masks.<n>We propose a Metadata-Guided Diffusion Model (M-GDM) to tackle these challenges.
  arXiv  Detail & Related papers  (2026-04-15T14:15:07Z)
• BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encoder [13.008131372229458]
  BackdoorIDS is a zero-shot, inference-time backdoor samples detection method for pretrained vision encoders.<n>It consistently outperforms existing defenses across diverse attack types, datasets, and model families.<n>BackdoorIDS is compatible with a wide range of encoder architectures, including CNNs, ViTs, CLIP, and LLaVA-1.5.
  arXiv  Detail & Related papers  (2026-03-12T08:32:19Z)
• StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models [57.674757786328236]
  StructSAM is a resolution-preserving merge-unmerge framework tailored to Segment Anything Model (SAM)<n>We show that StructSAM reduces encoder FLOPs by 25-30% (up to 40%+ with prompt-aware merging) with minor drops in mIoU/Dice.<n>We also provide a spectral graph coarsening view showing that score-guided merging yields bounded Laplacian spectral distortion compared to random or window-restricted baselines.
  arXiv  Detail & Related papers  (2026-03-07T18:30:58Z)
• Two Frames Matter: A Temporal Attack for Text-to-Video Model Jailbreaking [3.1471354211996956]
  Recent text-to-video (T2V) models can synthesize complex videos from lightweight natural language prompts.<n>Prior jailbreak attacks typically rewrite unsafe prompts into paraphrases that evade content filters.<n>In this paper, we identify a temporal trajectory infilling vulnerability of T2V systems under fragmented prompts.
  arXiv  Detail & Related papers  (2026-03-07T04:21:54Z)
• Detector-in-the-Loop Tracking: Active Memory Rectification for Stable Glottic Opening Localization [25.593059138669844]
  We propose a detector-in-the-loop framework that supervises Segment Anything Model 2(SAM2) through confidence-aligned state decisions and active memory rectification.<n>On emergency intubation videos, CL-MC achieves state-of-the-art performance, significantly reducing drift and missing rate.
  arXiv  Detail & Related papers  (2026-02-22T23:29:28Z)
• SPIRIT: Adapting Vision Foundation Models for Unified Single- and Multi-Frame Infrared Small Target Detection [18.86422994684341]
  Infrared small target detection (IRSTD) is crucial for surveillance and early-warning, with deployments spanning both single-frame analysis and video-mode tracking.<n>We propose SPIRIT, a unified and VFM-compatible framework that adapts VFMs to IRSTD via lightweight physics-informed plug-ins.
  arXiv  Detail & Related papers  (2026-02-02T09:15:29Z)
• RedVisor: Reasoning-Aware Prompt Injection Defense via Zero-Copy KV Cache Reuse [47.85771791033142]
  We propose RedVisor, a framework that synthesizes the explainability of detection systems with the seamless integration of prevention strategies.<n>RedVisor is the first approach to leverage fine-grained reasoning paths to simultaneously detect attacks and guide the model's safe response.<n> Experiments demonstrate that RedVisor outperforms state-of-the-art defenses in detection accuracy and throughput while incurring negligible utility loss.
  arXiv  Detail & Related papers  (2026-02-02T08:26:51Z)
• MDiff4STR: Mask Diffusion Model for Scene Text Recognition [59.79818820650126]
  Mask Diffusion Models (MDMs) have emerged as a promising alternative to auto-regressive models (ARMs) for vision-language tasks.<n>We show that vanilla MDM lags behind ARMs in terms of accuracy, although it improves recognition efficiency.<n>We propose MDiff4STR, a Mask Diffusion model enhanced with two key improvement strategies tailored for Scene Text Recognition.
  arXiv  Detail & Related papers  (2025-12-01T08:57:51Z)
• RLBind: Adversarial-Invariant Cross-Modal Alignment for Unified Robust Embeddings [0.47097887983573417]
  Unified multi-modal encoders that bind vision, audio, and other sensors into a shared embedding space are attractive building blocks for robot perception and decision-making.<n>We introduce RLBind, a two-stage adversarial-invariant cross-modal alignment framework for robust unified embeddings.<n> RLBind consistently outperforms the LanguageBind backbone and standard fine-tuning baselines in both clean accuracy and norm-bounded adversarial robustness.
  arXiv  Detail & Related papers  (2025-09-17T19:35:52Z)
• VFM-Guided Semi-Supervised Detection Transformer under Source-Free Constraints for Remote Sensing Object Detection [9.029534000674388]
  VG-DETR integrates a Vision Foundation Model (VFM) into the training pipeline in a "free lunch" manner.<n>We introduce a VFM-guided pseudo-label mining strategy that leverages the VFM's semantic priors to assess the reliability of the generated pseudo-labels.<n>In addition, a dual-level VFM-guided alignment method is proposed, which aligns detector features with VFM embeddings at both the instance and image levels.
  arXiv  Detail & Related papers  (2025-08-15T02:35:56Z)
• ForensicsSAM: Toward Robust and Unified Image Forgery Detection and Localization Resisting to Adversarial Attack [56.0056378072843]
  We show that highly transferable adversarial images can be crafted solely via the upstream model.<n>We propose ForensicsSAM, a unified IFDL framework with built-in adversarial robustness.
  arXiv  Detail & Related papers  (2025-08-10T16:03:44Z)
• Proactive Disentangled Modeling of Trigger-Object Pairings for Backdoor Defense [0.0]
  Deep neural networks (DNNs) and generative AI (GenAI) are increasingly vulnerable to backdoor attacks.<n>In this paper, we introduce DBOM, a proactive framework that leverages structured disentanglement to identify and neutralize both seen and unseen backdoor threats.<n>We show that DBOM robustly detects poisoned images prior to downstream training, significantly enhancing the security of training pipelines.
  arXiv  Detail & Related papers  (2025-08-03T21:58:15Z)
• SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMs [57.880467106470775]
  Attackers can inject imperceptible perturbations into the training data, causing the model to generate malicious, attacker-controlled captions.<n>We propose Semantic Reward Defense (SRD), a reinforcement learning framework that mitigates backdoor behavior without prior knowledge of triggers.<n>SRD uses a Deep Q-Network to learn policies for applying discrete perturbations to sensitive image regions, aiming to disrupt the activation of malicious pathways.
  arXiv  Detail & Related papers  (2025-06-05T08:22:24Z)
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning [76.82121084745785]
  Believe Your Eyes (BYE) is a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples.<n>It achieves near-zero attack success rates while maintaining clean-task performance.
  arXiv  Detail & Related papers  (2025-05-22T17:11:58Z)
• Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models [70.03122709795122]
  Backdoor attacks targeting text-to-image diffusion models have advanced rapidly.<n>Current backdoor samples often exhibit two key abnormalities compared to benign samples.<n>We propose Trigger without Trace (TwT) by explicitly mitigating these consistencies.
  arXiv  Detail & Related papers  (2025-03-22T10:41:46Z)
• Mask Propagation for Efficient Video Semantic Segmentation [63.09523058489429]
  Video Semantic baseline degradation (VSS) involves assigning a semantic label to each pixel in a video sequence. We propose an efficient mask propagation framework for VSS, called SSSS. Our framework reduces up to 4x FLOPs compared to the per-frame Mask2Former with only up to 2% mIoU on the Cityscapes validation set.
  arXiv  Detail & Related papers  (2023-10-29T09:55:28Z)
• Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation [109.1912721224697]
  We present the Unified Frequency-Assisted transFormer framework, named UFAFormer, to address the DGM4 problem. By leveraging the discrete wavelet transform, we decompose images into several frequency sub-bands, capturing rich face forgery artifacts. Our proposed frequency encoder, incorporating intra-band and inter-band self-attentions, explicitly aggregates forgery features within and across diverse sub-bands.
  arXiv  Detail & Related papers  (2023-09-18T11:06:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.