Privacy-Aware In-Context Learning for Large Language Models

• URL: http://arxiv.org/abs/2509.13625v3
• Date: Tue, 23 Sep 2025 02:40:24 GMT
• Title: Privacy-Aware In-Context Learning for Large Language Models
• Authors: Bishnu Bhusal, Manoj Acharya, Ramneet Kaur, Colin Samplawski, Anirban Roy, Adam D. Cobb, Rohit Chadha, Susmit Jha,
• Abstract summary: Large language models (LLMs) raise privacy concerns due to potential exposure of sensitive information.<n>We introduce a novel private prediction framework for generating high-quality synthetic text with strong privacy guarantees.
• Score: 12.605629953620495
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have significantly transformed natural language understanding and generation, but they raise privacy concerns due to potential exposure of sensitive information. Studies have highlighted the risk of information leakage, where adversaries can extract sensitive information embedded in the prompts. In this work, we introduce a novel private prediction framework for generating high-quality synthetic text with strong privacy guarantees. Our approach leverages the Differential Privacy (DP) framework to ensure worst-case theoretical bounds on information leakage without requiring any fine-tuning of the underlying models. The proposed method performs inference on private records and aggregates the resulting per-token output distributions. This enables the generation of longer and coherent synthetic text while maintaining privacy guarantees. Additionally, we propose a simple blending operation that combines private and public inference to further enhance utility. Empirical evaluations demonstrate that our approach outperforms previous state-of-the-art methods on in-context-learning (ICL) tasks, making it a promising direction for privacy-preserving text generation while maintaining high utility.

Related papers

• Private PoEtry: Private In-Context Learning via Product of Experts [58.496468062236225]
  In-context learning (ICL) enables Large Language Models to adapt to new tasks with only a small set of examples at inference time.<n>Existing differential privacy approaches to ICL are either computationally expensive or rely on oversampling, synthetic data generation, or unnecessary thresholding.<n>We reformulate private ICL through the lens of a Product-of-Experts model. This gives a theoretically grounded framework, and the algorithm can be trivially parallelized.<n>We find that our method improves accuracy by more than 30 percentage points on average compared to prior DP-ICL methods, while maintaining strong privacy guarantees.
  arXiv  Detail & Related papers  (2026-02-04T19:56:24Z)
• Zero-Shot Privacy-Aware Text Rewriting via Iterative Tree Search [60.197239728279534]
  Large language models (LLMs) in cloud-based services have raised significant privacy concerns.<n>Existing text anonymization and de-identification techniques, such as rule-based redaction and scrubbing, often struggle to balance privacy preservation with text naturalness and utility.<n>We propose a zero-shot, tree-search-based iterative sentence rewriting algorithm that systematically obfuscates or deletes private information while preserving coherence, relevance, and naturalness.
  arXiv  Detail & Related papers  (2025-09-25T07:23:52Z)
• The Double-edged Sword of LLM-based Data Reconstruction: Understanding and Mitigating Contextual Vulnerability in Word-level Differential Privacy Text Sanitization [53.51921540246166]
  We show that Language Large Models (LLMs) can exploit the contextual vulnerability of DP-sanitized texts.<n>Experiments uncover a double-edged sword effect of LLM reconstructions on privacy and utility.<n>We propose recommendations for using data reconstruction as a post-processing step.
  arXiv  Detail & Related papers  (2025-08-26T12:22:45Z)
• RL-Finetuned LLMs for Privacy-Preserving Synthetic Rewriting [17.294176570269]
  We propose a reinforcement learning framework that fine-tunes a large language model (LLM) using a composite reward function.<n>The privacy reward combines semantic cues with structural patterns derived from a minimum spanning tree (MST) over latent representations.<n> Empirical results show that the proposed method significantly enhances author obfuscation and privacy metrics without degrading semantic quality.
  arXiv  Detail & Related papers  (2025-08-25T04:38:19Z)
• Token-Level Privacy in Large Language Models [7.4143291213663955]
  We introduce dchi-stencil, a novel token-level privacy-preserving mechanism that integrates contextual and semantic information.<n>By incorporating both semantic and contextual nuances, dchi-stencil achieves a robust balance between privacy and utility.<n>This work highlights the potential of dchi-stencil to set a new standard for privacy-preserving NLP in modern, high-risk applications.
  arXiv  Detail & Related papers  (2025-03-05T16:27:25Z)
• NAP^2: A Benchmark for Naturalness and Privacy-Preserving Text Rewriting by Learning from Human [56.46355425175232]
  We suggest sanitizing sensitive text using two common strategies used by humans.<n>We curate the first corpus, coined NAP2, through both crowdsourcing and the use of large language models.<n>Compared to the prior works on anonymization, the human-inspired approaches result in more natural rewrites.
  arXiv  Detail & Related papers  (2024-06-06T05:07:44Z)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [82.7042006247124]
  We show that even the most capable AI models reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
  arXiv  Detail & Related papers  (2023-10-27T04:15:30Z)
• InferDPT: Privacy-Preserving Inference for Black-box Large Language Model [66.07752875835506]
  InferDPT is the first practical framework for the privacy-preserving Inference of black-box LLMs.<n>RANTEXT is a novel differential privacy mechanism integrated into the perturbation module of InferDPT.
  arXiv  Detail & Related papers  (2023-10-18T18:00:11Z)
• PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners [81.571305826793]
  We introduce Contextual Privacy Protection Language Models (PrivacyMind) Our work offers a theoretical analysis for model design and benchmarks various techniques. In particular, instruction tuning with both positive and negative examples stands out as a promising method.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe [32.63295550058343]
  We show that a simple and practical recipe in the text domain is effective in generating useful synthetic text with strong privacy protection. Our method produces synthetic text that is competitive in terms of utility with its non-private counterpart.
  arXiv  Detail & Related papers  (2022-10-25T21:21:17Z)
• CAPE: Context-Aware Private Embeddings for Private Language Learning [0.5156484100374058]
  Context-Aware Private Embeddings (CAPE) is a novel approach which preserves privacy during training of embeddings. CAPE applies calibrated noise through differential privacy, preserving the encoded semantic links while obscuring sensitive information. Experimental results demonstrate that the proposed approach reduces private information leakage better than either single intervention.
  arXiv  Detail & Related papers  (2021-08-27T14:50:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.