Membership Inference Attacks for Retrieval Based In-Context Learning for Document Question Answering

• URL: http://arxiv.org/abs/2605.04116v1
• Date: Tue, 05 May 2026 08:19:17 GMT
• Title: Membership Inference Attacks for Retrieval Based In-Context Learning for Document Question Answering
• Abstract summary: We show that remotely hosted applications employing in-context learning can be vulnerable to membership-inference attacks.<n>We propose two black-box membership inference attacks that exploit query text prefixes to distinguish member from non-member inputs.
• Score: 5.932575574212546
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We show that remotely hosted applications employing in-context learning when augmented with a retrieval function to select in-context examples can be vulnerable to membership-inference attacks even when the service provider and users are separate parties. We propose two black-box membership inference attacks that exploit query text prefixes to distinguish member from non-member inputs. The first attack uses a reference model to estimate an otherwise unavailable loss metric. The second attack improves upon it by eliminating the reference model and instead computing a membership statistic through a simple but novel weighted-averaging scheme. Our comprehensive empirical evaluations consider a stricter case in which the adversary has a paraphrased version of the text in the queries and show that our attacks can exhibit stronger resilience to paraphrasing and outperform three prior attacks in many cases with small number of prefixes. We also adapt an existing ensemble prompting defense to our setting, demonstrating that it substantially mitigates the privacy leakage caused by our second attack.

Related papers

• Automatic Red Teaming for Implicit Vulnerabilities of Text-to-Image Models [58.59008920408855]
  We propose Adversarial Probing for Implicit VulnErabilities (AdvPIE) to expose implicit vulnerabilities without requiring access to the parameters of target models.<n> AdvPIE adopts a policy agent to generate and refine implicit adversarial prompts based on the feedback from a judge agent.<n>Experiments on standard and safety-aligned T2I models show that AdvPIE1 effectively uncovers implicit vulnerabilities.
  arXiv  Detail & Related papers  (2026-09-05T13:45:09Z)
• Boundary-targeted Membership Inference Attacks on Safety Classifiers [70.20833439671131]
  Safety classifiers are trained on sensitive datasets including discussions of self-harm and mental health.<n>We introduce a new boundary-targeted selection strategy that identifies low confidence examples.<n>Experiments show that an adversary can recover 19% of the conversations a safety classifier flagged as indicating user distress, at a 5% false-positive rate.<n>This is $3.5$ times more than attacking using state-of-the-art MIA methods alone.
  arXiv  Detail & Related papers  (2026-05-21T12:05:22Z)
• Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning [24.914116986408327]
  We propose PromptMIA, a membership inference attack tailored to federated prompt-tuning.<n>We empirically show that PromptMIA consistently attains high advantage in this game across diverse benchmark datasets.<n>The results highlight non-trivial challenges for current defenses and offer insights into their limitations.
  arXiv  Detail & Related papers  (2026-01-10T17:50:05Z)
• Illusions of Relevance: Using Content Injection Attacks to Deceive Retrievers, Rerankers, and LLM Judges [52.96987928118327]
  We find that embedding models for retrieval, rerankers, and large language model (LLM) relevance judges are vulnerable to content injection attacks.<n>We identify two primary threats: (1) inserting unrelated or harmful content within passages that still appear deceptively "relevant", and (2) inserting entire queries or key query terms into passages to boost their perceived relevance.<n>Our study systematically examines the factors that influence an attack's success, such as the placement of injected content and the balance between relevant and non-relevant material.
  arXiv  Detail & Related papers  (2025-01-30T18:02:15Z)
• Quantifying Privacy Risks of Prompts in Visual Prompt Learning [33.244695390157595]
  We perform the first comprehensive privacy assessment of prompts learned by visual prompt learning. Our empirical evaluation shows that the prompts are vulnerable to both attacks. We show that membership inference attacks against prompts can be successful with relaxed adversarial assumptions.
  arXiv  Detail & Related papers  (2023-10-18T13:51:27Z)
• SLMIA-SR: Speaker-Level Membership Inference Attacks against Speaker Recognition Systems [6.057334150052503]
  SLMIA-SR is the first membership inference attack tailored to speaker recognition (SR) Our attack is versatile and can work in both white-box and black-box scenarios.
  arXiv  Detail & Related papers  (2023-09-14T18:40:28Z)
• Generalizable Black-Box Adversarial Attack with Meta Learning [54.196613395045595]
  In black-box adversarial attack, the target model's parameters are unknown, and the attacker aims to find a successful perturbation based on query feedback under a query budget. We propose to utilize the feedback information across historical attacks, dubbed example-level adversarial transferability. The proposed framework with the two types of adversarial transferability can be naturally combined with any off-the-shelf query-based attack methods to boost their performance.
  arXiv  Detail & Related papers  (2023-01-01T07:24:12Z)
• Preserving Semantics in Textual Adversarial Attacks [0.0]
  Up to 70% of adversarial examples generated by adversarial attacks should be discarded because they do not preserve semantics. We propose a new, fully supervised sentence embedding technique called Semantics-Preserving-Encoder (SPE) Our method outperforms existing sentence encoders used in adversarial attacks by achieving 1.2x - 5.1x better real attack success rate.
  arXiv  Detail & Related papers  (2022-11-08T12:40:07Z)
• A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks [72.7373468905418]
  We develop an open-source toolkit OpenBackdoor to foster the implementations and evaluations of textual backdoor learning. We also propose CUBE, a simple yet strong clustering-based defense baseline.
  arXiv  Detail & Related papers  (2022-06-17T02:29:23Z)
• Adversarial Text Normalization [2.9434930072968584]
  Adversarial Text Normalizer restores baseline performance on attacked content with low computational overhead. We find that text normalization provides a task-agnostic defense against character-level attacks.
  arXiv  Detail & Related papers  (2022-06-08T19:44:03Z)
• Zero-Query Transfer Attacks on Context-Aware Object Detectors [95.18656036716972]
  Adversarial attacks perturb images such that a deep neural network produces incorrect classification results. A promising approach to defend against adversarial attacks on natural multi-object scenes is to impose a context-consistency check. We present the first approach for generating context-consistent adversarial attacks that can evade the context-consistency check.
  arXiv  Detail & Related papers  (2022-03-29T04:33:06Z)
• Sampling Attacks: Amplification of Membership Inference Attacks by Repeated Queries [74.59376038272661]
  We introduce sampling attack, a novel membership inference technique that unlike other standard membership adversaries is able to work under severe restriction of no access to scores of the victim model. We show that a victim model that only publishes the labels is still susceptible to sampling attacks and the adversary can recover up to 100% of its performance. For defense, we choose differential privacy in the form of gradient perturbation during the training of the victim model as well as output perturbation at prediction time.
  arXiv  Detail & Related papers  (2020-09-01T12:54:54Z)
• Revisiting Membership Inference Under Realistic Assumptions [87.13552321332988]
  We study membership inference in settings where some of the assumptions typically used in previous research are relaxed. This setting is more realistic than the balanced prior setting typically considered by researchers. We develop a new inference attack based on the intuition that inputs corresponding to training set members will be near a local minimum in the loss function.
  arXiv  Detail & Related papers  (2020-05-21T20:17:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.