Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs

• URL: http://arxiv.org/abs/2501.19287v1
• Date: Fri, 31 Jan 2025 16:48:38 GMT
• Title: Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs
• Authors: James Flemings, Haosheng Gan, Hongyi Li, Meisam Razaviyayn, Murali Annavaram,
• Abstract summary: In-context learning (ICL) can be improved by augmenting prompts with relevant input-output examples (demonstrations) ICL demonstrations can contain privacy-sensitive information, which can be leaked and/or regurgitated by the LLM output. We propose $textttdps-mozo$, a decoding framework that generates DP text by sampling from the product of multiple one-shot outputs mixed with a zero-shot output.
• Score: 13.790550802100842
• License:
• Abstract: In-context learning (ICL) has shown promising improvement in downstream task adaptation of LLMs by augmenting prompts with relevant input-output examples (demonstrations). However, the ICL demonstrations can contain privacy-sensitive information, which can be leaked and/or regurgitated by the LLM output. Differential Privacy (DP), a widely adopted privacy safeguard, has emerged to mitigate this privacy leakage, with recent work demonstrating strong privacy-utility tradeoffs in classification tasks for ICL. However, generation tasks for ICL are challenging due to the high-dimensional output space of open-ended generation. To this end, we propose $\texttt{dps-mozo}$, Differentially Private Sampling by Mixing One-shot with Zero-shot Outputs, a decoding framework that generates DP text by sampling from the product of multiple one-shot outputs mixed with a zero-shot output. This mixing effectively reduces the amount of information that can be leaked by each demonstration. By utilizing the inherent randomness in sampling from the mixed distributions, we can achieve DP without adding noise, thereby improving the privacy-utility tradeoff. Our experimental evaluations show $\texttt{dps-mozo}$ can achieve a strong privacy guarantee, $\epsilon=2$, with minimal utility degradation compared to non-private few-shot learning, $\textbf{0.3}$% ROUGE-L F1 score decrease on the SAMSum dataset with Gemma 2 2B.

Related papers

• Differentially Private Steering for Large Language Model Alignment [55.30573701583768]
  We present the first study of aligning Large Language Models with private datasets. Our work proposes the textitunderlinePrivate underlineSteering for LLM underlineAment (PSA) algorithm. Our results show that PSA achieves DP guarantees for LLM alignment with minimal loss in performance.
  arXiv  Detail & Related papers  (2025-01-30T17:58:36Z)
• Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output [49.893971654861424]
  We present a light-weight approach for detecting nonfactual outputs from retrieval-augmented generation (RAG) We compute a factuality score that can be thresholded to yield a binary decision. Our experiments show high area under the ROC curve (AUC) across a wide range of relevant open source datasets.
  arXiv  Detail & Related papers  (2024-11-01T20:44:59Z)
• Adaptively Private Next-Token Prediction of Large Language Models [13.297381972044558]
  We introduce a noisy screening mechanism that filters out queries with potentially expensive privacy loss. AdaPMixED can reduce the privacy loss by 16x while preserving the utility over the original PMixED.
  arXiv  Detail & Related papers  (2024-10-02T20:34:24Z)
• $\mathbb{USCD}$: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decoding [64.00025564372095]
  Large language models (LLMs) have shown remarkable capabilities in code generation. The effects of hallucinations (e.g., output noise) make it challenging for LLMs to generate high-quality code in one pass. We propose a simple and effective textbfuncertainty-aware textbfselective textbfcontrastive textbfdecoding.
  arXiv  Detail & Related papers  (2024-09-09T02:07:41Z)
• Entropy Law: The Story Behind Data Compression and LLM Performance [115.70395740286422]
  We find that model performance is negatively correlated to the compression ratio of training data, which usually yields a lower training loss. Based on the findings of the entropy law, we propose a quite efficient and universal data selection method. We also present an interesting application of entropy law that can detect potential performance risks at the beginning of model training.
  arXiv  Detail & Related papers  (2024-07-09T08:14:29Z)
• Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation [37.55812121348268]
  In-context learning (ICL) with large language models (LLMs) on private datasets poses privacy risks. We propose a novel algorithm that generates synthetic few-shot demonstrations from the private dataset with formal differential privacy guarantees.
  arXiv  Detail & Related papers  (2023-09-21T03:59:00Z)
• Privacy Amplification via Shuffling: Unified, Simplified, and Tightened [20.10078781197001]
  We propose a comprehensive framework for privacy amplification in both single-message and multi-message shuffle protocols. Our theoretical results demonstrate that our framework provides tighter bounds, especially for local randomizers with extremal probability design. Our bounds also result in a remarkably efficient $tildeO(n)$ algorithm that numerically amplifies privacy in less than $10$ seconds for $n=108$ users.
  arXiv  Detail & Related papers  (2023-04-11T06:27:25Z)
• RDP-GAN: A R\'enyi-Differential Privacy based Generative Adversarial Network [75.81653258081435]
  Generative adversarial network (GAN) has attracted increasing attention recently owing to its impressive ability to generate realistic samples with high privacy protection. However, when GANs are applied on sensitive or private training examples, such as medical or financial records, it is still probable to divulge individuals' sensitive and private information. We propose a R'enyi-differentially private-GAN (RDP-GAN), which achieves differential privacy (DP) in a GAN by carefully adding random noises on the value of the loss function during training.
  arXiv  Detail & Related papers  (2020-07-04T09:51:02Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.