Whispers in Grammars: Injecting Covert Backdoors to Compromise Dense Retrieval Systems

• URL: http://arxiv.org/abs/2402.13532v2
• Date: Tue, 17 Dec 2024 06:54:32 GMT
• Title: Whispers in Grammars: Injecting Covert Backdoors to Compromise Dense Retrieval Systems
• Authors: Quanyu Long, Yue Deng, LeiLei Gan, Wenya Wang, Sinno Jialin Pan,
• Abstract summary: This paper investigates a novel attack scenario where the attackers aim to mislead the retrieval system into retrieving the attacker-specified contents.<n>Those contents, injected into the retrieval corpus by attackers, can include harmful text like hate speech or spam.<n>Unlike prior methods that rely on model weights and generate conspicuous, unnatural outputs, we propose a covert backdoor attack triggered by grammar errors.
• Score: 40.131588857153275
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dense retrieval systems have been widely used in various NLP applications. However, their vulnerabilities to potential attacks have been underexplored. This paper investigates a novel attack scenario where the attackers aim to mislead the retrieval system into retrieving the attacker-specified contents. Those contents, injected into the retrieval corpus by attackers, can include harmful text like hate speech or spam. Unlike prior methods that rely on model weights and generate conspicuous, unnatural outputs, we propose a covert backdoor attack triggered by grammar errors. Our approach ensures that the attacked models can function normally for standard queries while covertly triggering the retrieval of the attacker's contents in response to minor linguistic mistakes. Specifically, dense retrievers are trained with contrastive loss and hard negative sampling. Surprisingly, our findings demonstrate that contrastive loss is notably sensitive to grammatical errors, and hard negative sampling can exacerbate susceptibility to backdoor attacks. Our proposed method achieves a high attack success rate with a minimal corpus poisoning rate of only 0.048%, while preserving normal retrieval performance. This indicates that the method has negligible impact on user experience for error-free queries. Furthermore, evaluations across three real-world defense strategies reveal that the malicious passages embedded within the corpus remain highly resistant to detection and filtering, underscoring the robustness and subtlety of the proposed attack.

Related papers

• Illusions of Relevance: Using Content Injection Attacks to Deceive Retrievers, Rerankers, and LLM Judges [52.96987928118327]
  We find that embedding models for retrieval, rerankers, and large language model (LLM) relevance judges are vulnerable to content injection attacks. We identify two primary threats: (1) inserting unrelated or harmful content within passages that still appear deceptively "relevant", and (2) inserting entire queries or key query terms into passages to boost their perceived relevance. Our study systematically examines the factors that influence an attack's success, such as the placement of injected content and the balance between relevant and non-relevant material.
  arXiv  Detail & Related papers  (2025-01-30T18:02:15Z)
• Document Screenshot Retrievers are Vulnerable to Pixel Poisoning Attacks [72.4498910775871]
  Vision-language model (VLM)-based retrievers leverage document screenshots embedded as vectors to enable effective search and offer a simplified pipeline over traditional text-only methods. In this study, we propose three pixel poisoning attack methods designed to compromise VLM-based retrievers.
  arXiv  Detail & Related papers  (2025-01-28T12:40:37Z)
• Backdoored Retrievers for Prompt Injection Attacks on Retrieval Augmented Generation of Large Language Models [0.0]
  Retrieval Augmented Generation (RAG) addresses this issue by combining Large Language Models with up-to-date information retrieval. This paper investigates prompt injection attacks on RAG, focusing on malicious objectives beyond misinformation. We build upon existing corpus poisoning techniques and propose a novel backdoor attack aimed at the fine-tuning process of the dense retriever component.
  arXiv  Detail & Related papers  (2024-10-18T14:02:34Z)
• Can We Trust the Unlabeled Target Data? Towards Backdoor Attack and Defense on Model Adaptation [120.42853706967188]
  We explore the potential backdoor attacks on model adaptation launched by well-designed poisoning target data. We propose a plug-and-play method named MixAdapt, combining it with existing adaptation algorithms.
  arXiv  Detail & Related papers  (2024-01-11T16:42:10Z)
• Punctuation Matters! Stealthy Backdoor Attack for Language Models [36.91297828347229]
  A backdoored model produces normal outputs on the clean samples while performing improperly on the texts. Some attack methods even cause grammatical issues or change the semantic meaning of the original texts. We propose a novel stealthy backdoor attack method against textual models, which is called textbfPuncAttack.
  arXiv  Detail & Related papers  (2023-12-26T03:26:20Z)
• Model Stealing Attack against Recommender System [85.1927483219819]
  Some adversarial attacks have achieved model stealing attacks against recommender systems. In this paper, we constrain the volume of available target data and queries and utilize auxiliary data, which shares the item set with the target data, to promote model stealing attacks.
  arXiv  Detail & Related papers  (2023-12-18T05:28:02Z)
• Poisoning Retrieval Corpora by Injecting Adversarial Passages [79.14287273842878]
  We propose a novel attack for dense retrieval systems in which a malicious user generates a small number of adversarial passages. When these adversarial passages are inserted into a large retrieval corpus, we show that this attack is highly effective in fooling these systems. We also benchmark and compare a range of state-of-the-art dense retrievers, both unsupervised and supervised.
  arXiv  Detail & Related papers  (2023-10-29T21:13:31Z)
• Large Language Models Are Better Adversaries: Exploring Generative Clean-Label Backdoor Attacks Against Text Classifiers [25.94356063000699]
  Backdoor attacks manipulate model predictions by inserting innocuous triggers into training and test data. We focus on more realistic and more challenging clean-label attacks where the adversarial training examples are correctly labeled. Our attack, LLMBkd, leverages language models to automatically insert diverse style-based triggers into texts.
  arXiv  Detail & Related papers  (2023-10-28T06:11:07Z)
• Mitigating Backdoor Poisoning Attacks through the Lens of Spurious Correlation [43.75579468533781]
  backdoors can be implanted through crafting training instances with a specific trigger and a target label. This paper posits that backdoor poisoning attacks exhibit emphspurious correlation between simple text features and classification labels. Our empirical study reveals that the malicious triggers are highly correlated to their target labels.
  arXiv  Detail & Related papers  (2023-05-19T11:18:20Z)
• Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models [41.1058288041033]
  We propose ProAttack, a novel and efficient method for performing clean-label backdoor attacks based on the prompt. Our method does not require external triggers and ensures correct labeling of poisoned samples, improving the stealthy nature of the backdoor attack.
  arXiv  Detail & Related papers  (2023-05-02T06:19:36Z)
• Verifying the Robustness of Automatic Credibility Assessment [50.55687778699995]
  We show that meaning-preserving changes in input text can mislead the models. We also introduce BODEGA: a benchmark for testing both victim models and attack methods on misinformation detection tasks. Our experimental results show that modern large language models are often more vulnerable to attacks than previous, smaller solutions.
  arXiv  Detail & Related papers  (2023-03-14T16:11:47Z)
• Detecting Backdoors in Deep Text Classifiers [43.36440869257781]
  We present the first robust defence mechanism that generalizes to several backdoor attacks against text classification models. Our technique is highly accurate at defending against state-of-the-art backdoor attacks, including data poisoning and weight poisoning.
  arXiv  Detail & Related papers  (2022-10-11T07:48:03Z)
• Zero-Query Transfer Attacks on Context-Aware Object Detectors [95.18656036716972]
  Adversarial attacks perturb images such that a deep neural network produces incorrect classification results. A promising approach to defend against adversarial attacks on natural multi-object scenes is to impose a context-consistency check. We present the first approach for generating context-consistent adversarial attacks that can evade the context-consistency check.
  arXiv  Detail & Related papers  (2022-03-29T04:33:06Z)
• Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learning [95.60856995067083]
  This work is among the first to perform adversarial defense for ASV without knowing the specific attack algorithms. We propose to perform adversarial defense from two perspectives: 1) adversarial perturbation purification and 2) adversarial perturbation detection. Experimental results show that our detection module effectively shields the ASV by detecting adversarial samples with an accuracy of around 80%.
  arXiv  Detail & Related papers  (2021-06-01T07:10:54Z)
• Backdoor Attack against Speaker Verification [86.43395230456339]
  We show that it is possible to inject the hidden backdoor for infecting speaker verification models by poisoning the training data. We also demonstrate that existing backdoor attacks cannot be directly adopted in attacking speaker verification.
  arXiv  Detail & Related papers  (2020-10-22T11:10:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.