Differentially Private Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2602.14374v1
• Date: Mon, 16 Feb 2026 00:52:57 GMT
• Title: Differentially Private Retrieval-Augmented Generation
• Authors: Tingting Tang, James Flemings, Yongqin Wang, Murali Annavaram,
• Abstract summary: Retrieval-augmented generation (RAG) is a widely used framework for reducing hallucinations in large language models (LLMs)<n>RAG poses serious privacy risks when the database contains sensitive corpora, such as medical records or legal documents.<n>We present DP-KSA, a novel privacy-preserving RAG algorithm that integrates DP using the propose-test-release paradigm.
• Score: 13.622078883013442
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-augmented generation (RAG) is a widely used framework for reducing hallucinations in large language models (LLMs) on domain-specific tasks by retrieving relevant documents from a database to support accurate responses. However, when the database contains sensitive corpora, such as medical records or legal documents, RAG poses serious privacy risks by potentially exposing private information through its outputs. Prior work has demonstrated that one can practically craft adversarial prompts that force an LLM to regurgitate the augmented contexts. A promising direction is to integrate differential privacy (DP), a privacy notion that offers strong formal guarantees, into RAG systems. However, naively applying DP mechanisms into existing systems often leads to significant utility degradation. Particularly for RAG systems, DP can reduce the usefulness of the augmented contexts leading to increase risk of hallucination from the LLMs. Motivated by these challenges, we present DP-KSA, a novel privacy-preserving RAG algorithm that integrates DP using the propose-test-release paradigm. DP-KSA follows from a key observation that most question-answering (QA) queries can be sufficiently answered with a few keywords. Hence, DP-KSA first obtains an ensemble of relevant contexts, each of which will be used to generate a response from an LLM. We utilize these responses to obtain the most frequent keywords in a differentially private manner. Lastly, the keywords are augmented into the prompt for the final output. This approach effectively compresses the semantic space while preserving both utility and privacy. We formally show that DP-KSA provides formal DP guarantees on the generated output with respect to the RAG database. We evaluate DP-KSA on two QA benchmarks using three instruction-tuned LLMs, and our empirical results demonstrate that DP-KSA achieves a strong privacy-utility tradeoff.

Related papers

• Generation-Augmented Generation: A Plug-and-Play Framework for Private Knowledge Injection in Large Language Models [48.65910216527897]
  Generation-Augmented Generation (GAG) treats private expertise as an additional expert modality and injects it via a compact representation-level interface.<n>GAG improves specialist performance over strong RAG baselines by 15.34% and 14.86% on two benchmarks.
  arXiv  Detail & Related papers  (2026-01-13T04:23:36Z)
• SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systems [53.51921540246166]
  Retrieval-Augmented Generation (RAG) techniques have become widely popular.<n>RAG involves the coupling of Large Language Models (LLMs) with domain-specific knowledge bases.<n>The proliferation of RAG has sparked concerns about data privacy.
  arXiv  Detail & Related papers  (2026-01-07T14:50:41Z)
• Private-RAG: Answering Multiple Queries with LLMs while Keeping Your Data Private [21.980739918403344]
  Retrieval-augmented generation (RAG) enhances large language models (LLMs) by retrieving documents from an external corpus at inference time.<n>When this corpus contains sensitive information, unprotected RAG systems are at risk of leaking private information.<n>In this paper, we study the more practical multi-query setting and propose two DP-RAG algorithms.
  arXiv  Detail & Related papers  (2025-11-10T21:12:32Z)
• Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG) [13.736991294264827]
  We propose DP-SynRAG, a framework that uses LLMs to generate differentially private synthetic RAG databases.<n>Unlike prior methods, the synthetic text can be reused once created, thereby avoiding repeated noise injection and additional privacy costs.<n>Experiments show that DP-SynRAG achieves superior performanec to the state-of-the-art private RAG systems while maintaining a fixed privacy budget.
  arXiv  Detail & Related papers  (2025-10-08T07:15:50Z)
• Fine-Grained Privacy Extraction from Retrieval-Augmented Generation Systems via Knowledge Asymmetry Exploitation [15.985529058573912]
  Retrieval-augmented generation (RAG) systems enhance large language models (LLMs) by integrating external knowledge bases.<n>Existing privacy attacks on RAG systems can trigger data leakage but often fail to accurately isolate knowledge-base-derived sentences within mixed responses.<n>This paper presents a novel black-box attack framework that exploits knowledge asymmetry between RAG and standard LLMs to achieve fine-grained privacy extraction.
  arXiv  Detail & Related papers  (2025-07-31T03:50:16Z)
• Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs [69.10441885629787]
  Retrieval-Augmented Generation (RAG) lifts the factuality of Large Language Models (LLMs) by injecting external knowledge.<n>It falls short on problems that demand multi-step inference; conversely, purely reasoning-oriented approaches often hallucinate or mis-ground facts.<n>This survey synthesizes both strands under a unified reasoning-retrieval perspective.
  arXiv  Detail & Related papers  (2025-07-13T03:29:41Z)
• Differentially Private Relational Learning with Entity-level Privacy Guarantees [17.567309430451616]
  This work presents a principled framework for relational learning with formal entity-level DP guarantees.<n>We introduce an adaptive gradient clipping scheme that modulates clipping thresholds based on entity occurrence frequency.<n>These contributions lead to a tailored DP-SGD variant for relational data with provable privacy guarantees.
  arXiv  Detail & Related papers  (2025-06-10T02:03:43Z)
• UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities [53.76854299076118]
  UniversalRAG is a novel RAG framework designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities.<n>We propose a modality-aware routing mechanism that dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it.<n>We validate UniversalRAG on 8 benchmarks spanning multiple modalities, showing its superiority over various modality-specific and unified baselines.
  arXiv  Detail & Related papers  (2025-04-29T13:18:58Z)
• DP-GTR: Differentially Private Prompt Protection via Group Text Rewriting [25.526993224085093]
  Existing methods primarily focus on document-level rewriting, neglecting the rich, multi-granular representations of text.<n>We introduce DP-GTR, a novel three-stage framework that leverages local differential privacy (DP) and the composition theorem via group text rewriting.<n>Our framework is compatible with existing rewriting techniques, serving as a plug-in to enhance privacy protection.
  arXiv  Detail & Related papers  (2025-03-06T21:39:42Z)
• Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning [62.224804688233]
  differential privacy (DP) offers a promising solution by ensuring models are 'almost indistinguishable' with or without any particular privacy unit. We study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users.
  arXiv  Detail & Related papers  (2024-06-20T13:54:32Z)
• How Private are DP-SGD Implementations? [61.19794019914523]
  We show that there can be a substantial gap between the privacy analysis when using the two types of batch sampling. Our result shows that there can be a substantial gap between the privacy analysis when using the two types of batch sampling.
  arXiv  Detail & Related papers  (2024-03-26T13:02:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.