VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data

• URL: http://arxiv.org/abs/2410.00296v1
• Date: Tue, 1 Oct 2024 00:37:29 GMT
• Title: VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data
• Authors: Xuefeng Du, Reshmi Ghosh, Robert Sim, Ahmed Salem, Vitor Carvalho, Emily Lawton, Yixuan Li, Jack W. Stokes,
• Abstract summary: VLMGuard is a novel learning framework that leverages the unlabeled user prompts in the wild for malicious prompt detection. We present an automated maliciousness estimation score for distinguishing between benign and malicious samples. Our framework does not require extra human annotations, offering strong flexibility and practicality for real-world applications.
• Score: 29.806775884883685
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) are essential for contextual understanding of both visual and textual information. However, their vulnerability to adversarially manipulated inputs presents significant risks, leading to compromised outputs and raising concerns about the reliability in VLM-integrated applications. Detecting these malicious prompts is thus crucial for maintaining trust in VLM generations. A major challenge in developing a safeguarding prompt classifier is the lack of a large amount of labeled benign and malicious data. To address the issue, we introduce VLMGuard, a novel learning framework that leverages the unlabeled user prompts in the wild for malicious prompt detection. These unlabeled prompts, which naturally arise when VLMs are deployed in the open world, consist of both benign and malicious information. To harness the unlabeled data, we present an automated maliciousness estimation score for distinguishing between benign and malicious samples within this unlabeled mixture, thereby enabling the training of a binary prompt classifier on top. Notably, our framework does not require extra human annotations, offering strong flexibility and practicality for real-world applications. Extensive experiment shows VLMGuard achieves superior detection results, significantly outperforming state-of-the-art methods. Disclaimer: This paper may contain offensive examples; reader discretion is advised.

Related papers

• Breaking the Prompt Wall (I): A Real-World Case Study of Attacking ChatGPT via Lightweight Prompt Injection [12.565784666173277]
  This report presents a real-world case study demonstrating how prompt injection can attack large language model platforms such as ChatGPT. We show how adversarial prompts can be injected via user inputs, web-based retrieval, and system-level agent instructions.
  arXiv  Detail & Related papers  (2025-04-20T05:59:00Z)
• Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-tuning [23.71517734919702]
  Vision-language models (VLMs) have made remarkable strides in generative modeling with multimodal inputs. Current alignment strategies rely on supervised safety fine-tuning with curated datasets. We show that supervised fine-tuning inadvertently reinforces spurious correlations between superficial textual patterns and safety responses.
  arXiv  Detail & Related papers  (2025-03-14T19:52:08Z)
• Retention Score: Quantifying Jailbreak Risks for Vision Language Models [60.48306899271866]
  Vision-Language Models (VLMs) are integrated with Large Language Models (LLMs) to enhance multi-modal machine learning capabilities. This paper aims to assess the resilience of VLMs against jailbreak attacks that can compromise model safety compliance and result in harmful outputs. To evaluate a VLM's ability to maintain its robustness against adversarial input perturbations, we propose a novel metric called the textbfRetention Score.
  arXiv  Detail & Related papers  (2024-12-23T13:05:51Z)
• Aligning LLMs to Be Robust Against Prompt Injection [55.07562650579068]
  We show that alignment can be a powerful tool to make LLMs more robust against prompt injection attacks. Our method -- SecAlign -- first builds an alignment dataset by simulating prompt injection attacks. Our experiments show that SecAlign robustifies the LLM substantially with a negligible hurt on model utility.
  arXiv  Detail & Related papers  (2024-10-07T19:34:35Z)
• HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router [42.222681564769076]
  We introduce HiddenGuard, a novel framework for fine-grained, safe generation in Large Language Models. HiddenGuard incorporates Prism, which operates alongside the LLM to enable real-time, token-level detection and redaction of harmful content. Our experiments demonstrate that HiddenGuard achieves over 90% in F1 score for detecting and redacting harmful content.
  arXiv  Detail & Related papers  (2024-10-03T17:10:41Z)
• ANVIL: Anomaly-based Vulnerability Identification without Labelled Training Data [5.938113434208745]
  Supervised learning-based software vulnerability detectors often fall short due to the inadequate availability of labelled training data. In this paper, we explore a different approach by reframing vulnerability detection as one of anomaly detection. Our approach achieves $1.62times$ to $2.18times$ better Top-5 accuracies and $1.02times$ to $1.29times$ times better ROC scores on line-level vulnerability detection tasks.
  arXiv  Detail & Related papers  (2024-08-28T03:28:17Z)
• The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models [31.166994121531232]
  Vision-Language Models (VLMs) combine visual and textual understanding, rendering them well-suited for diverse tasks. These capabilities are built upon training on large amount of uncurated data crawled from the web. In this paper, we assess whether these vulnerabilities exist, focusing on identity leakage.
  arXiv  Detail & Related papers  (2024-08-02T12:36:13Z)
• MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection [107.15164718585666]
  We investigate the root cause of VLMs' biased prediction under the open vocabulary detection context. Our observations lead to a simple yet effective paradigm, coded MarvelOVD, that generates significantly better training targets. Our method outperforms the other state-of-the-arts by significant margins.
  arXiv  Detail & Related papers  (2024-07-31T09:23:57Z)
• Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models [79.0183835295533]
  We introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities. Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content. We propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings.
  arXiv  Detail & Related papers  (2023-12-21T01:08:39Z)
• Prompt Highlighter: Interactive Control for Multi-Modal LLMs [50.830448437285355]
  This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. We introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. We find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs.
  arXiv  Detail & Related papers  (2023-12-07T13:53:29Z)
• Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection [70.28425745910711]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following. This capability brings with it the risk of prompt injection attacks. We evaluate the robustness of instruction-following LLMs against such attacks.
  arXiv  Detail & Related papers  (2023-08-17T06:21:50Z)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection [64.67495502772866]
  Large Language Models (LLMs) are increasingly being integrated into various applications. We show how attackers can override original instructions and employed controls using Prompt Injection attacks. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities.
  arXiv  Detail & Related papers  (2023-02-23T17:14:38Z)
• Unlearnable Clusters: Towards Label-agnostic Unlearnable Examples [128.25509832644025]
  There is a growing interest in developing unlearnable examples (UEs) against visual privacy leaks on the Internet. UEs are training samples added with invisible but unlearnable noise, which have been found can prevent unauthorized training of machine learning models. We present a novel technique called Unlearnable Clusters (UCs) to generate label-agnostic unlearnable examples with cluster-wise perturbations.
  arXiv  Detail & Related papers  (2022-12-31T04:26:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.