Comparative Analysis of Inference-Time Defense Methods for Multimodal Large Language Models

• URL: http://arxiv.org/abs/2606.10904v1
• Date: Tue, 09 Jun 2026 14:13:54 GMT
• Title: Comparative Analysis of Inference-Time Defense Methods for Multimodal Large Language Models
• Abstract summary: Multimodal large language models (MLLMs) now appear in safety-critical applications.<n>Retraining a model for each new vulnerability class is usually too expensive to be practical.<n>We report a comparative empirical evaluation of three-time defense methods and their combinations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language models (MLLMs) now appear in safety-critical applications, but the visual channel leaves them open to adversarial attacks that predominantly text-oriented safety alignment addresses only in part. Retraining a model for each new vulnerability class is usually too expensive to be practical. We report a comparative empirical evaluation of three inference-time defense methods and their combinations, run on eight models from the InternVL and Qwen-VL families across seven safety benchmarks that span four attack classes and total 9,000 evaluation samples. Every figure below comes from the same unified proxy classifier. Five findings emerge from the evaluation. First, within the evaluated models and benchmarks, no single defense dominates across all settings: what works depends on the model's baseline safety and on the attack type. Second, combining defenses directly drives benign-query over-refusal to 97-100% across all eight evaluated models, and SmoothVLM on its own reaches 99.2-100%. Third, a simple safety prompt keeps utility largely intact (0.0-18.2% over-refusal across all eight models, five of them below 7%, although two exceeded 15%) while still yielding moderate safety gains. Fourth, different attack classes expose different weaknesses across the evaluated setup, which is why multi-benchmark evaluation matters. Fifth, in a preliminary whitebox test on two models (n=20), text-level defenses suppressed a PGD visual attack that had succeeded without any defense: the defenses act at the output stage, where gradient optimization has limited direct leverage in the tested configuration. Read together, these results argue for adaptive defense selection rather than a single fixed defense configuration.

Related papers

• Evaluating Prompting-Based Defenses Against Domain-Camouflaged Injection Attacks [0.0]
  Domain-camouflaged injection attacks embed malicious instructions in retrieved content using domain-appropriate vocabulary.<n>We evaluate five prompting-based defenses against domain-camouflaged injection across three model families.<n>Paraphrasing retrieved content before agent processing is the most consistently effective defense in this benchmark.
  arXiv  Detail & Related papers  (2026-06-16T22:51:02Z)
• Automated jailbreak attack targeting multiple defense strategies [11.710908982928594]
  Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks.<n>However, their safety remains a critical concern due to their susceptibility to adversarial prompt-based attacks.<n>We present UNIATTACK, an adversarial testing framework designed from a defense-oriented perspective.
  arXiv  Detail & Related papers  (2026-06-15T14:09:37Z)
• Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks [0.3303672705634661]
  We evaluate whether frontier LLMs are ready for cybersecurity through a dual-mode benchmark.<n>We test six frontier models (GPT-5.4, Codex5.3, Claude Opus4.6, Sonnet4.6, Gemini3.1Pro and Gemini3Flash) and two domain-specialized models across four testing paradigms.
  arXiv  Detail & Related papers  (2026-05-22T05:24:43Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model [50.29667251847595]
  We show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models.<n>We propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space.<n>In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2% in DAN, 31.3% in WildJailbreak and 35.4 % in StrongREJECT benchmarks.
  arXiv  Detail & Related papers  (2026-04-01T02:42:41Z)
• Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks [0.0]
  Large Language Model (LLM) safety guardrail models have emerged as a primary defense mechanism against harmful content generation.<n>This study evaluated ten publicly available guardrail models from Meta, Google, IBM, NVIDIA, Alibaba, and Allen AI across 1,445 test prompts spanning 21 attack categories.
  arXiv  Detail & Related papers  (2025-11-27T03:01:09Z)
• Safeguarding Efficacy in Large Language Models: Evaluating Resistance to Human-Written and Algorithmic Adversarial Prompts [0.0]
  This paper presents a systematic security assessment of four prominent Large Language Models (LLMs) against adversarial attack vectors.<n>We evaluate Phi-2, Llama-2-7B-Chat, GPT-3.5-Turbo, and GPT-4 across four distinct attack categories: human-written prompts, AutoDAN, Greedy Coordinate Gradient (GCG), and Tree-of-Attacks-with-pruning (TAP)
  arXiv  Detail & Related papers  (2025-10-12T21:48:34Z)
• Evaluating the Robustness of the "Ensemble Everything Everywhere" Defense [90.7494670101357]
  Ensemble everything everywhere is a defense to adversarial examples.<n>We show that this defense is not robust to adversarial attack.<n>We then use standard adaptive attack techniques to reduce the defense's robust accuracy.
  arXiv  Detail & Related papers  (2024-11-22T10:17:32Z)
• TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models [53.91006249339802]
  We propose a novel defense method called Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness of CLIP against visual adversarial attacks. TAPT is a test-time defense method that learns defensive bimodal (textual and visual) prompts to robustify the inference process of CLIP. We evaluate the effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other zero-shot datasets.
  arXiv  Detail & Related papers  (2024-11-20T08:58:59Z)
• MultiRobustBench: Benchmarking Robustness Against Multiple Attacks [86.70417016955459]
  We present the first unified framework for considering multiple attacks against machine learning (ML) models. Our framework is able to model different levels of learner's knowledge about the test-time adversary. We evaluate the performance of 16 defended models for robustness against a set of 9 different attack types.
  arXiv  Detail & Related papers  (2023-02-21T20:26:39Z)
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks [65.20660287833537]
  In this paper we propose two extensions of the PGD-attack overcoming failures due to suboptimal step size and problems of the objective function. We then combine our novel attacks with two complementary existing ones to form a parameter-free, computationally affordable and user-independent ensemble of attacks to test adversarial robustness.
  arXiv  Detail & Related papers  (2020-03-03T18:15:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.