TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models

• URL: http://arxiv.org/abs/2308.03906v1
• Date: Mon, 7 Aug 2023 20:48:07 GMT
• Title: TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models
• Abstract summary: We present a Multimodal Backdoor Defense technique TIJO (Trigger Inversion using Joint Optimization) Recent work arXiv:2112.07668 has demonstrated successful backdoor attacks on multimodal models for the Visual Question Answering task. We propose TIJO that defends against dual-key attacks through a joint optimization that reverse-engineers the trigger in both the image and text modalities.
• Score: 18.485290653242853
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We present a Multimodal Backdoor Defense technique TIJO (Trigger Inversion using Joint Optimization). Recent work arXiv:2112.07668 has demonstrated successful backdoor attacks on multimodal models for the Visual Question Answering task. Their dual-key backdoor trigger is split across two modalities (image and text), such that the backdoor is activated if and only if the trigger is present in both modalities. We propose TIJO that defends against dual-key attacks through a joint optimization that reverse-engineers the trigger in both the image and text modalities. This joint optimization is challenging in multimodal models due to the disconnected nature of the visual pipeline which consists of an offline feature extractor, whose output is then fused with the text using a fusion module. The key insight enabling the joint optimization in TIJO is that the trigger inversion needs to be carried out in the object detection box feature space as opposed to the pixel space. We demonstrate the effectiveness of our method on the TrojVQA benchmark, where TIJO improves upon the state-of-the-art unimodal methods from an AUC of 0.6 to 0.92 on multimodal dual-key backdoors. Furthermore, our method also improves upon the unimodal baselines on unimodal backdoors. We present ablation studies and qualitative results to provide insights into our algorithm such as the critical importance of overlaying the inverted feature triggers on all visual features during trigger inversion. The prototype implementation of TIJO is available at https://github.com/SRI-CSL/TIJO.

Related papers

• Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model [73.59026525350808]
  We propose a dual adversarial fine-tuning framework that jointly optimize visual and semantic supervision signals from two modalities.<n>Our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering tasks.
  arXiv  Detail & Related papers  (2026-07-21T10:49:29Z)
• From Parameters to Feature Space: Task Arithmetic for Backdoor Mitigation in Model Merging [9.772291868457812]
  Model merging (MM) has gained significant attention as a cost-effective approach to integrate multiple task-specific models into a unified model.<n>Recent work reveals that MM is highly susceptible to backdoor attacks.<n>We propose Linear Feature Path Minimization (LFPM), a backdoor mitigation framework for model merging.
  arXiv  Detail & Related papers  (2026-06-10T13:51:36Z)
• Cross-Modal Backdoors in Multimodal Large Language Models [21.893043087583667]
  A single compromised connector can establish a reusable latent-space backdoor pathway across modalities.<n>The attack achieves up to 99.9% attack success rate (ASR) in same-modality settings.
  arXiv  Detail & Related papers  (2026-05-08T09:29:50Z)
• Scaling Exposes the Trigger: Input-Level Backdoor Detection in Text-to-Image Diffusion Models via Cross-Attention Scaling [12.573852448122716]
  Text-to-image (T2I) diffusion models have achieved remarkable success in image synthesis, but their reliance on large-scale data and open ecosystems introduces serious backdoor security risks.<n>Existing defenses, particularly input-level methods, are more practical for deployment but often rely on observable anomalies that become unreliable under stealthy, semantics-preserving trigger designs.<n>We propose SET, an input-level backdoor detection framework that constructs response-offset features under multi-scale perturbations and learns a compact benign response space from a small set of clean samples.
  arXiv  Detail & Related papers  (2026-04-14T08:31:37Z)
• GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection [19.80016468034245]
  GateFusion is a novel architecture that combines strong pretrained unimodal encoders with a Hierarchical Gated Fusion Decoder (HiGate)<n>HiGate enables progressive, multi-depth fusion by adaptively injecting contextual features from one modality into the other at multiple layers of the Transformer backbone.
  arXiv  Detail & Related papers  (2025-12-17T18:56:52Z)
• Patronus: Identifying and Mitigating Transferable Backdoors in Pre-trained Language Models [20.691302472834675]
  Transferable backdoors pose a severe threat to the Pre-trained Language Models (PLMs) supply chain.<n>We propose Patronus, a novel framework that use input-side invariance of triggers against parameter shifts.<n>Experiments demonstrate that Patronus achieves $geq98.7%$ backdoor detection recall and reduce attack success rates to clean settings.
  arXiv  Detail & Related papers  (2025-12-07T15:51:56Z)
• BackdoorVLM: A Benchmark for Backdoor Attacks on Vision-Language Models [63.5775877701015]
  We introduce textbfBackdoorVLM, the first comprehensive benchmark for evaluating backdoor attacks on vision-language models (VLMs)<n>BackdoorVLM organizes multimodal backdoor threats into 5 representative categories: targeted refusal, malicious injection, jailbreak, concept substitution, and perceptual hijack.<n>We evaluate these threats using 12 representative attack methods spanning text, image, and bimodal triggers, tested on 2 open-source VLMs and 3 multimodal datasets.
  arXiv  Detail & Related papers  (2025-11-24T09:30:38Z)
• MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generation [86.82285754460491]
  We propose a new benchmark designed to evaluate both text and image output modalities.<n>This performance degradation is strongly correlated with poor alignment between the generated reasoning and the final image.<n>We propose a parallel multimodal diffusion framework, MMaDA-Parallel, that enables continuous, bidirectional interaction between text and images.
  arXiv  Detail & Related papers  (2025-11-12T18:58:21Z)
• ShortcutBreaker: Low-Rank Noisy Bottleneck with Global Perturbation Attention for Multi-Class Unsupervised Anomaly Detection [59.89803740308262]
  ShortcutBreaker is a novel unified feature-reconstruction framework for MUAD tasks.<n>It features two key innovations to address the issue of shortcuts.<n>The proposed method achieves a remarkable image-level AUROC of 99.8%, 98.9%, 90.6%, and 87.8% on four datasets.
  arXiv  Detail & Related papers  (2025-10-21T06:51:30Z)
• EAM: Enhancing Anything with Diffusion Transformers for Blind Super-Resolution [11.331361804059625]
  Enhancing Anything Model (EAM) is a novel Blind Super-Resolution method.<n>We introduce a novel block, $Psi$-DiT, which effectively guides the DiT to enhance image restoration.<n>EAM achieves state-of-the-art results across multiple datasets, outperforming existing methods in both quantitative metrics and visual quality.
  arXiv  Detail & Related papers  (2025-05-08T13:03:07Z)
• MLLM-Enhanced Face Forgery Detection: A Vision-Language Fusion Solution [36.79921476565535]
  VLF-FFD is a novel Vision-Language Fusion solution for MLLM-enhanced Face Forgery Detection.<n> EFF++ is a frame-level, explainability-driven extension of the widely used FaceForensics++ dataset.<n>VLF-FFD achieves state-of-the-art (SOTA) performance in both cross-dataset and intra-dataset evaluations.
  arXiv  Detail & Related papers  (2025-05-04T06:58:21Z)
• DSwinIR: Rethinking Window-based Attention for Image Restoration [109.38288333994407]
  We propose the Deformable Sliding Window Transformer (DSwinIR) as a new foundational backbone architecture for image restoration.<n>At the heart of DSwinIR is the proposed novel Deformable Sliding Window (DSwin) Attention.<n>Extensive experiments show that DSwinIR sets a new state-of-the-art across a wide spectrum of image restoration tasks.
  arXiv  Detail & Related papers  (2025-04-07T09:24:41Z)
• Efficient Input-level Backdoor Defense on Text-to-Image Synthesis via Neuron Activation Variation [37.075824084492524]
  NaviT2I is an efficient input-level backdoor defense framework against diverse T2I backdoors.<n>Our approach is based on the new observation that trigger tokens tend to induce significant neuron activation variation in the early stage of the diffusion generation process.
  arXiv  Detail & Related papers  (2025-03-09T05:27:44Z)
• DADM: Dual Alignment of Domain and Modality for Face Anti-spoofing [58.62312400472865]
  Multi-modal face anti-spoofing (FAS) has emerged as a prominent research focus.<n>We propose a alignment module between modalities based on mutual information.<n>We employ a dual alignment optimization method that aligns both sub-domain hyperplanes and modality angle margins.
  arXiv  Detail & Related papers  (2025-03-01T10:12:00Z)
• Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion [13.696706205837238]
  Pre-trained multi-modal Vision-Language Models like CLIP are widely used off-the-shelf for a variety of applications. We argue that this is inherently due to the CLIP-style inter-modal contrastive loss that does not enforce any intra-modal constraints. We empirically show that, in the intra-modal tasks of image-to-image and text-to-text retrieval, approaching these tasks inter-modally significantly improves performance.
  arXiv  Detail & Related papers  (2025-02-06T17:58:59Z)
• Combinational Backdoor Attack against Customized Text-to-Image Models [29.08082452640628]
  We propose a textbfCombinational textbfBackdoor textbfAttack against textbfCustomized textbfT2I models (CBACT2I)<n>Different from previous backdoor attacks against T2I models, CBACT2I embeds the backdoor into the text encoder and the conditional diffusion model separately.
  arXiv  Detail & Related papers  (2024-11-19T10:20:31Z)
• BadCM: Invisible Backdoor Attack Against Cross-Modal Learning [110.37205323355695]
  We introduce a novel bilateral backdoor to fill in the missing pieces of the puzzle in the cross-modal backdoor. BadCM is the first invisible backdoor method deliberately designed for diverse cross-modal attacks within one unified framework.
  arXiv  Detail & Related papers  (2024-10-03T03:51:53Z)
• TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoors [36.07978634674072]
  Diffusion models are vulnerable to backdoor attacks that compromise their integrity. We propose TERD, a backdoor defense framework that builds unified modeling for current attacks. TERD secures a 100% True Positive Rate (TPR) and True Negative Rate (TNR) across datasets of varying resolutions.
  arXiv  Detail & Related papers  (2024-09-09T03:02:16Z)
• T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models [70.03122709795122]
  We propose a comprehensive defense method named T2IShield to detect, localize, and mitigate backdoor attacks. We find the "Assimilation Phenomenon" on the cross-attention maps caused by the backdoor trigger. For backdoor sample detection, T2IShield achieves a detection F1 score of 88.9$%$ with low computational cost.
  arXiv  Detail & Related papers  (2024-07-05T01:53:21Z)
• GraFT: Gradual Fusion Transformer for Multimodal Re-Identification [0.8999666725996975]
  We introduce the textbfGradual Fusion Transformer (GraFT) for multimodal ReID. GraFT employs learnable fusion tokens that guide self-attention across encoders, adeptly capturing both modality-specific and object-specific features. We demonstrate these enhancements through extensive ablation studies and show that GraFT consistently surpasses established multimodal ReID benchmarks.
  arXiv  Detail & Related papers  (2023-10-25T00:15:40Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers [58.802352477207094]
  We explore the great potential of a pre-trained vision Transformer (ViT) to bridge the vast distribution gap between two modalities. We propose a mask modeling strategy that randomly masks a specific modality of some tokens to enforce the interaction between tokens from different modalities interacting proactively. Experiments demonstrate that our plug-and-play training augmentation techniques can significantly boost state-of-the-art one-stream and two trackersstream to a large extent in terms of both tracking precision and success rate.
  arXiv  Detail & Related papers  (2023-07-09T08:58:47Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)
• Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing [19.142582966452935]
  We investigate three key factors (i.e., inputs, pre-training, and finetuning) in ViT for multimodal FAS with RGB, Infrared (IR), and Depth. We propose the modality-asymmetric masked autoencoder (M$2$A$2$E) for multimodal FAS self-supervised pre-training without costly annotated labels.
  arXiv  Detail & Related papers  (2023-02-11T17:02:34Z)
• Dual-Key Multimodal Backdoors for Visual Question Answering [26.988750557552983]
  We show that multimodal networks are vulnerable to a novel type of attack that we refer to as Dual-Key Multimodal Backdoors. This attack exploits the complex fusion mechanisms used by state-of-the-art networks to embed backdoors that are both effective and stealthy. We present an extensive study of multimodal backdoors on the Visual Question Answering (VQA) task with multiple architectures and visual feature backbones.
  arXiv  Detail & Related papers  (2021-12-14T18:59:52Z)
• Adaptive Context-Aware Multi-Modal Network for Depth Completion [107.15344488719322]
  We propose to adopt the graph propagation to capture the observed spatial contexts. We then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively. Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively. Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks.
  arXiv  Detail & Related papers  (2020-08-25T06:00:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.