Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailment

• URL: http://arxiv.org/abs/2605.24312v2
• Date: Sun, 31 May 2026 02:22:17 GMT
• Title: Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailment
• Abstract summary: We present MEntA (Membership Entailment Attack), a query-efficient MIA that leverages natural-language entailment to maximize information gained per query.<n>Across NFCorpus, SCIDOCS, and TREC-COVID, MEntA achieves up to 0.991 AUC with only 5 queries, outperforming prior methods by up to 0.42 AUC under equivalent conditions.<n>Our findings expose the feasibility of realistic, low-cost privacy leakage in RAG systems and highlight the urgent need for privacy-aware retrieval and defense mechanisms.
• Score: 19.95305571949895
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-augmented generation (RAG) has become central to large language model (LLM) deployments, grounding responses in enterprise or proprietary data to reduce hallucinations. However, this design introduces a new privacy risk: model outputs may signal the presence of specific documents in the retrieval corpus, enabling membership inference attacks (MIAs) that leak sensitive information. Existing MIAs are feasible, but they often rely on easily detected templated queries or require many non-templated yet costly and repetitive queries, limiting practicality. We ask: Can an adversary launch a limited-budget, surrogate-free, stealthy, and defense-agnostic membership inference attack using non-templated queries? We present MEntA (Membership Entailment Attack), a query-efficient MIA that leverages natural-language entailment to maximize information gained per query. By asking low-cost, broad, information-seeking questions and measuring entailment between model responses and candidate documents, MEntA eliminates the need for costly shadow models and large query budgets. Across NFCorpus, SCIDOCS, and TREC-COVID, MEntA achieves up to 0.991 AUC with only 5 queries, outperforming prior methods by up to 0.42 AUC under equivalent conditions. It remains effective under state-of-the-art (SOTA) RAG defenses, while current detectors either miss MEntA or flag benign queries at high rates. Regarding cost, MEntA reduces total attack cost by up to 65 $\times$ lower compared to SOTA attacks under the same attack setting. Our findings expose the feasibility of realistic, low-cost privacy leakage in RAG systems and highlight the urgent need for privacy-aware retrieval and defense mechanisms.

Related papers

• MIRAGE: Defending Long-Form RAG Against Misinformation Pollution [87.40537931342082]
  We introduce MIRAGE, a training-free, model-agnostic defense for long-form RAG.<n>MIRAGE builds an NLI-based cross-document claim graph and applies a Defended-Claims Gate to either condition generation on a consistent, multi-source supported subset or to block retrieval and answer parametrically.
  arXiv  Detail & Related papers  (2026-07-06T13:36:12Z)
• PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say [48.84133320567554]
  LLM-based agents are rapidly advancing, autonomously invoking external tools to complete tasks for users.<n> Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context.<n>We introduce emphPrivacyPeek, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents.
  arXiv  Detail & Related papers  (2026-05-29T04:55:12Z)
• WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agents [53.84430233754606]
  We introduce WMAttack, an automated attack-search framework for adversarial evaluation of world-model agents.<n> WMAttack formulates robustness evaluation as a finite-budget search over attack configurations.<n>We show that WMAttack consistently discovers stronger attacks than the evaluated baselines.
  arXiv  Detail & Related papers  (2026-05-22T04:31:09Z)
• Adaptive Defense Orchestration for RAG: A Sentinel-Strategist Architecture against Multi-Vector Attacks [0.0]
  Retrieval-augmented generation systems are increasingly deployed in sensitive domains such as healthcare and law.<n>This capability introduces significant security risks, including membership inference, data poisoning, and unintended content leakage.<n>We propose the Sentinel-Strategist architecture, a context-aware framework for risk analysis and defense selection.
  arXiv  Detail & Related papers  (2026-04-22T11:17:10Z)
• ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Querying [18.047579533992252]
  We propose ADAM, a novel privacy attack that features data distribution estimation of a victim agent's memory.<n>Our attack substantially outperforms state-of-the-art ones, achieving up to 100% ASRs.
  arXiv  Detail & Related papers  (2026-04-10T07:22:11Z)
• LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications [51.56484100374058]
  We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow.<n>The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract.<n>We evaluate the harness on ticket-routing and BEIR tasks with full Azure matrix coverage.
  arXiv  Detail & Related papers  (2026-03-28T18:03:32Z)
• LoMime: Query-Efficient Membership Inference using Model Extraction in Label-Only Settings [3.4228148164859857]
  Membership inference attacks (MIAs) threaten the privacy of machine learning models by revealing whether a specific data point was used during training.<n>We propose a cost-effective label-only MIA framework based on transferability and model extraction.<n>Our method matches the performance of state-of-the-art label-only MIAs while significantly reducing query costs.
  arXiv  Detail & Related papers  (2026-02-21T18:57:17Z)
• NeuroFilter: Privacy Guardrails for Conversational LLM Agents [50.75206727081996]
  This work addresses the computational challenge of enforcing privacy for agentic Large Language Models (LLMs)<n>NeuroFilter is a guardrail framework that operationalizes contextual integrity by mapping norm violations to simple directions in the model's activation space.<n>A comprehensive evaluation across over 150,000 interactions, covering models from 7B to 70B parameters, illustrates the strong performance of NeuroFilter.
  arXiv  Detail & Related papers  (2026-01-21T05:16:50Z)
• BudgetLeak: Membership Inference Attacks on RAG Systems via the Generation Budget Side Channel [21.74370009270649]
  Membership inference attacks (MIAs) are a common technique to assess such risks.<n>We identify a previously unexplored side channel in RAG systems: the generation budget.<n>We propose BudgetLeak, a novel membership inference attack that probes responses under different budgets.
  arXiv  Detail & Related papers  (2025-11-15T05:25:04Z)
• Rescuing the Unpoisoned: Efficient Defense against Knowledge Corruption Attacks on RAG Systems [11.812488957698038]
  Large language models (LLMs) are reshaping numerous facets of our daily lives, leading widespread adoption as web-based services.<n>Retrieval-Augmented Generation (RAG) has emerged as a promising direction by generating responses grounded in external knowledge sources.<n>Recent studies demonstrate the vulnerability of RAG, such as knowledge corruption attacks by injecting misleading information.<n>In this work, we introduce RAGDefender, a resource-efficient defense mechanism against knowledge corruption.
  arXiv  Detail & Related papers  (2025-11-03T06:39:58Z)
• Benchmarking Misuse Mitigation Against Covert Adversaries [80.74502950627736]
  Existing language model safety evaluations focus on overt attacks and low-stakes tasks.<n>We develop Benchmarks for Stateful Defenses (BSD), a data generation pipeline that automates evaluations of covert attacks and corresponding defenses.<n>Our evaluations indicate that decomposition attacks are effective misuse enablers, and highlight stateful defenses as a countermeasure.
  arXiv  Detail & Related papers  (2025-06-06T17:33:33Z)
• Exploring the limits of strong membership inference attacks on large language models [70.49900359876595]
  State-of-the-art membership inference attacks (MIAs) typically require training many reference models.<n>We show that strong MIAs can succeed on pre-trained language models, but their effectiveness remains limited.
  arXiv  Detail & Related papers  (2025-05-24T16:23:43Z)
• mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA [78.45521005703958]
  multimodal Retrieval-Augmented Generation (mRAG) is naturally introduced to provide MLLMs with comprehensive and up-to-date knowledge. We propose a novel framework called textbfRetrieval-textbfReftextbfAugmented textbfGeneration (mR$2$AG) which achieves adaptive retrieval and useful information localization. mR$2$AG significantly outperforms state-of-the-art MLLMs on INFOSEEK and Encyclopedic-VQA
  arXiv  Detail & Related papers  (2024-11-22T16:15:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.