When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learning

• URL: http://arxiv.org/abs/2608.04052v1
• Date: Tue, 04 Aug 2026 10:20:41 GMT
• Title: When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learning
• Abstract summary: Backdoor attacks in multimodal contrastive learning (MCL) have garnered growing attention in recent years.<n>Existing detection-based defenses predominantly rely on the CLIPScore metric.<n>We introduce CASCADE, a novel two-stage Coarse-to-Fine Conformal Backdoor Detection framework.
• Score: 22.47689018895596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Backdoor attacks in multimodal contrastive learning (MCL) have garnered growing attention in recent years, as many downstream tasks critically depend on pre-trained MCL models. Existing detection-based defenses predominantly rely on the CLIPScore metric, under the assumption that poisoned pairs exhibit lower semantic similarity between the image and the caption. However, we identify two critical flaws remaining in existing methods: (1) the substantial overlap between CLIPScore distributions of benign and poisoned pairs undermines the reliability of this metric, and (2) fixed-threshold detection cannot provide statistical guarantees for ambiguous samples within overlapping regions. To overcome these limitations, we propose integrating conformal prediction (CP), a statistical framework that quantifies uncertainty through nonconformity scores (NCSs), to establish provable confidence bounds for detecting poisoned image-caption pairs. Building on CP, we introduce CASCADE, a novel two-stage Coarse-to-Fine Conformal Backdoor Detection framework. The coarse-grained stage uses cross-modality consistency to identify high-confidence benign and poisoned pairs. In the fine-grained stage, a reference set is constructed from high-confidence poisoned pairs, and instance-level NCSs based on text-space similarity are computed for each sample in the unidentified subset. These NCSs measure conformity to the poisoning distribution and enable precise identification of latent poisoned pairs within the unidentified subset. Extensive experiments on the large-scale CC3M dataset demonstrate that CASCADE achieves an average FPR of 5.79% at 100% TPR and an average AUROC of 0.9867 across diverse attacks, while remaining effective against adaptive attacks.

Related papers

• Fusing Spectral Signatures and Activation Clustering for Backdoor Detection in Healthcare Imaging Models: Method, Implementation, and Evaluation [0.0]
  This paper contributes a score-level fusion rule combining per-class spectral ranking with activation-clustering flags into a single per-sample poisoning score and a model-level agreement statistic.<n>An evaluation of that pipeline against synthetically poisoned variants of a public medical imaging benchmark and CIFAR-10 is presented.
  arXiv  Detail & Related papers  (2026-09-13T05:19:12Z)
• CADRE: Stable, Parameter Efficient Adaptation of Medical Vision Language Models with Bounded Forgetting and Prior Drift [0.10742675209112622]
  Updating a deployed model for a new imaging modality can fail silently in two ways that harm patients.<n>We study parameter-efficient continual adaptation through these two properties rather than forgetting leaderboard accuracy.<n>We present CADRE: a frozen-backbone framework combining low-rank adaptation (LoRA) with an online, self-scaling, similarity-aware elastic weight consolidation term.
  arXiv  Detail & Related papers  (2026-06-22T15:34:32Z)
• Post-hoc Selective Classification for Reliable Synthetic Image Detection [5.743750345138882]
  Deep neural network-based synthetic image detectors (SIDs) lack reliability in deployment.<n>We propose a selective classification (SC) strategy by allowing SIDs to abstain from making low confidence predictions.<n>We introduce a preference optimization algorithm that aggregates confidence scores extracted from different layers to a final confidence estimate.
  arXiv  Detail & Related papers  (2026-05-09T00:25:22Z)
• Quantifiable Uncertainty: A Stochastic Consensus Multi-Agent RAG Framework for Robust Malware Detection [1.2183405753834562]
  We present MAGMA, a framework that decouples malware analysis into semantic code retrieval and probabilistic verification.<n>We show that MAGMA achieves a 98.4% detection rate, substantially exceeding existing solutions.
  arXiv  Detail & Related papers  (2026-05-08T18:46:24Z)
• Towards Trustworthy Depression Estimation via Disentangled Evidential Learning [50.22167852149165]
  We propose EviDep, an evidential learning framework that jointly quantifies depression severity.<n>EviDep enforces strict information integrity to guarantee robust evidence synthesis.<n>It achieves state-of-the-art predictive accuracy and superior uncertainty calibration, delivering a robust fail-safe mechanism for trustworthy clinical screening.
  arXiv  Detail & Related papers  (2026-04-17T13:27:11Z)
• Probabilistic Feature Imputation and Uncertainty-Aware Multimodal Federated Aggregation [0.2283362795949404]
  Probabilistic Feature Imputation Network produces uncertainty estimates alongside imputed features.<n>Experiments on federated chest X-ray classification using CheXpert, NIH Open-I, and PadChest demonstrate consistent improvements over deterministic baselines.
  arXiv  Detail & Related papers  (2026-04-14T17:03:14Z)
• Neural Uncertainty Principle: A Unified View of Adversarial Fragility and LLM Hallucination [60.197429875410286]
  Adversarial vulnerability in vision and hallucination in large language models are conventionally viewed as separate problems.<n>Formalizing a Neural Uncertainty Principle (NUP) under a loss-induced state, we find that in near-bound regimes, further compression must be accompanied by increased sensitivity dispersion.<n>In vision, masking highly coupled components improves robustness without costly adversarial training.<n>In language, the same prefill-stage probe detects hallucination risk before generating any answer tokens.
  arXiv  Detail & Related papers  (2026-03-20T02:07:10Z)
• BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning [73.46118996284888]
  Research on backdoor attacks against multimodal contrastive learning models faces two key challenges: stealthiness and persistence.<n>We propose BadCLIP++, a unified framework that tackles both challenges.<n>For stealthiness, we introduce a semantic-fusion QR micro-trigger that embeds imperceptible patterns near task-relevant regions.<n>For persistence, we stabilize trigger embeddings via radius shrinkage and centroid alignment.
  arXiv  Detail & Related papers  (2026-02-19T08:31:16Z)
• The Eminence in Shadow: Exploiting Feature Boundary Ambiguity for Robust Backdoor Attacks [51.468144272905135]
  Deep neural networks (DNNs) underpin critical applications yet remain vulnerable to backdoor attacks.<n>We provide a theoretical analysis targeting backdoor attacks, focusing on how sparse decision boundaries enable disproportionate model manipulation.<n>We propose Eminence, an explainable and robust black-box backdoor framework with provable theoretical guarantees and inherent stealth properties.
  arXiv  Detail & Related papers  (2025-12-11T08:09:07Z)
• VALID: a Validated Algorithm for Learning in Decentralized Networks with Possible Adversarial Presence [13.612214163974459]
  We introduce the paradigm of validated decentralized learning for undirected networks with heterogeneous data. VALID protocol is the first to achieve a validated learning guarantee. Remarkably, VALID retains optimal performance metrics in adversary-free environments.
  arXiv  Detail & Related papers  (2024-05-12T15:55:43Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• Hierarchical Semi-Supervised Contrastive Learning for Contamination-Resistant Anomaly Detection [81.07346419422605]
  Anomaly detection aims at identifying deviant samples from the normal data distribution. Contrastive learning has provided a successful way to sample representation that enables effective discrimination on anomalies. We propose a novel hierarchical semi-supervised contrastive learning framework, for contamination-resistant anomaly detection.
  arXiv  Detail & Related papers  (2022-07-24T18:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.