Unifying Adversarially Robust Model Experts in Vision-Language Models

• URL: http://arxiv.org/abs/2607.27897v1
• Date: Thu, 30 Jul 2026 09:13:35 GMT
• Title: Unifying Adversarially Robust Model Experts in Vision-Language Models
• Abstract summary: Vision-language models (VLMs) are vulnerable to adversarial attacks.<n>We propose a collaborative adversarial fine-tuning framework: CARE.<n> CARE maintains multiple experts during training, enables knowledge exchange through embedding-space harmonization, and consolidates the learned knowledge into a single robust model.
• Score: 33.73696500471732
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs), such as CLIP, are vulnerable to adversarial attacks, posing a serious problem for real-life applications and deployment. Adversarial fine-tuning emerges as a prominent defense method; however, different fine-tuning strategies often produce specialized models with distinct robustness characteristics. Each fine-tuned model in turn thrives in some evaluation settings but falters on others, limiting their defensive capabilities. We refer to these specialized fine-tuned models as robust model experts and propose a collaborative adversarial fine-tuning framework: CARE - Collaborative Adversarial Robustness fine-tuning using Embedding alignment. CARE maintains multiple experts during training, enables knowledge exchange through embedding-space harmonization, and consolidates the learned knowledge into a single unified robust model. Experts benefit from one another while preserving their individual specializations, enabling the final model to inherit complementary robustness properties. In this paper, we demonstrate CARE on two different adversarial fine-tuning strategies with complementary robustness behaviors. Extensive experiments on classic image classification and downstream vision-language tasks display the effectiveness of our approach, with CARE being able to outperform individually learned model experts. The results suggest that collaborative learning across model experts is a promising direction for improving adversarial robustness.

Related papers

• Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model [73.59026525350808]
  We propose a dual adversarial fine-tuning framework that jointly optimize visual and semantic supervision signals from two modalities.<n>Our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering tasks.
  arXiv  Detail & Related papers  (2026-07-21T10:49:29Z)
• CPC-VAR:Continual Personalized and Compositional Generation in Visual Autoregressive Models [69.9266117908314]
  Visual autoregressive ( VAR) models have emerged as efficient paradigm for text-to-image generation.<n>Existing VAR-based personalization methods fail to accommodate evolving user demands.<n>We present the first systematic study of continual personalized generation in VAR models.
  arXiv  Detail & Related papers  (2026-05-19T12:18:15Z)
• Harnessing Consistency for Robust Test-Time LLM Ensemble [88.55393815158608]
  CoRE is a plug-and-play technique that harnesses model consistency for robust LLM ensemble.<n> Token-level consistency captures fine-grained disagreements by applying a low-pass filter to downweight uncertain tokens.<n>Model-level consistency models global agreement by promoting model outputs with high self-confidence.
  arXiv  Detail & Related papers  (2025-10-12T04:18:45Z)
• Synergistic Weak-Strong Collaboration by Aligning Preferences [53.47675666475273]
  Current Large Language Models (LLMs) excel in general reasoning yet struggle with specialized tasks requiring proprietary or domain-specific knowledge.<n>We propose a collaborative framework that pairs a specialized weak model with a general strong model.<n>We find that the collaboration significantly outperforms each model alone by leveraging complementary strengths.
  arXiv  Detail & Related papers  (2025-04-21T15:57:33Z)
• MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models [30.04163729936878]
  Meticulous Adrial Attack (MAA) fully exploit model-independent characteristics and vulnerabilities of individual samples.<n>MAA emphasizes fine-grained optimization of adversarial images by developing a novel resizing and sliding crop (RScrop) technique.
  arXiv  Detail & Related papers  (2025-02-12T02:53:27Z)
• Sustainable Self-evolution Adversarial Training [41.35034408227795]
  We propose a novel Sustainable Self-Evolution Adversarial Training (SSEAT) framework.<n>We introduce a continual adversarial defense pipeline to realize learning from various kinds of adversarial examples.<n>We also propose an adversarial data replay module to better select more diverse and key relearning data.
  arXiv  Detail & Related papers  (2024-12-03T08:41:11Z)
• ExAL: An Exploration Enhanced Adversarial Learning Algorithm [0.0]
  We propose a novel Exploration-enhanced Adversarial Learning Algorithm (ExAL) ExAL integrates exploration-driven mechanisms to discover perturbations that maximize impact on the model's decision boundary. We evaluate the performance of ExAL on the MNIST Handwritten Digits and Blended Malware datasets.
  arXiv  Detail & Related papers  (2024-11-24T15:37:29Z)
• Aligning Visual Contrastive learning models via Preference Optimization [0.9438963196770565]
  This paper introduces a novel method for training contrastive learning models using different Preference Optimization (PO) methods to break down complex concepts.<n>Our method systematically aligns model behavior with desired preferences, enhancing performance on the targeted task.<n>In particular, we focus on enhancing model robustness against typographic attacks and inductive biases, commonly seen in contrastive vision-language models like CLIP.
  arXiv  Detail & Related papers  (2024-11-12T08:14:54Z)
• Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models [9.762046320216005]
  Large vision models have been found vulnerable to adversarial examples, emphasizing the need for enhancing their adversarial robustness. Recent approaches propose robust fine-tuning methods, such as adversarial tuning of low-rank adaptation (LoRA) in large vision models, but they still struggle to match the accuracy of full parameter adversarial fine-tuning. We propose hyper adversarial tuning (HyperAT), which leverages shared defensive knowledge among different methods to improve model robustness efficiently and effectively simultaneously.
  arXiv  Detail & Related papers  (2024-10-08T12:05:01Z)
• Can collaborative learning be private, robust and scalable? [6.667150890634173]
  We investigate the effectiveness of combining differential privacy, model compression and adversarial training to improve the robustness of models against adversarial samples in train- and inference-time attacks. Our investigation provides a practical overview of various methods that allow one to achieve a competitive model performance, a significant reduction in model's size and an improved empirical adversarial robustness without a severe performance degradation.
  arXiv  Detail & Related papers  (2022-05-05T13:51:44Z)
• Clustering Effect of (Linearized) Adversarial Robust Models [60.25668525218051]
  We propose a novel understanding of adversarial robustness and apply it on more tasks including domain adaption and robustness boosting. Experimental evaluations demonstrate the rationality and superiority of our proposed clustering strategy.
  arXiv  Detail & Related papers  (2021-11-25T05:51:03Z)
• Boosting Adversarial Training with Hypersphere Embedding [53.75693100495097]
  Adversarial training is one of the most effective defenses against adversarial attacks for deep learning models. In this work, we advocate incorporating the hypersphere embedding mechanism into the AT procedure. We validate our methods under a wide range of adversarial attacks on the CIFAR-10 and ImageNet datasets.
  arXiv  Detail & Related papers  (2020-02-20T08:42:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.