On the Privacy of LLMs: An Ablation Study

• URL: http://arxiv.org/abs/2605.02255v1
• Date: Mon, 04 May 2026 06:06:41 GMT
• Title: On the Privacy of LLMs: An Ablation Study
• Abstract summary: Large language models (LLMs) are increasingly deployed in interactive and retrieval-augmented settings.<n>We introduce a unified threat model and notation, reproduce a representative set of privacy attacks, and conduct a structured ablation study.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly deployed in interactive and retrieval-augmented settings, raising significant privacy concerns. While attacks such as Membership Inference (MIA), Attribute Inference (AIA), Data Extraction (DEA), and Backdoor Attacks (BA) have been studied, they are typically analyzed in isolation, leaving a gap in understanding their behavior under common system factors. In this paper, we introduce a unified threat model and notation, reproduce a representative set of privacy attacks, and conduct a structured ablation study to evaluate the impact of key factors such as model architecture, scale, dataset characteristics, and retrieval configuration. Our analysis reveals clear differences across attack types. Membership inference attacks, particularly mask-based variants, exhibit strong and reliable signals, while backdoor attacks achieve consistently high success rates due to their trigger-based nature. In contrast, attribute inference and data extraction attacks remain more challenging, resulting in lower accuracy, yet they pose significant risks as they target sensitive personal information. Overall, these results highlight that privacy risks in LLM systems are highly context-dependent and driven by design choices, emphasizing the need for holistic evaluation and informed deployment practices.

Related papers

• Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation [50.87199039334856]
  Retrieval-Augmented Generation (RAG) has become a cornerstone of knowledge-intensive applications.<n>Recent studies show that knowledge-extraction attacks can recover sensitive knowledge-base content through maliciously crafted queries.<n>We introduce the first systematic benchmark for knowledge-extraction attacks on RAG systems.
  arXiv  Detail & Related papers  (2026-02-10T01:27:46Z)
• Differential Privacy: Gradient Leakage Attacks in Federated Learning Environments [0.6850683267295249]
  Federated Learning (FL) allows for the training of Machine Learning models in a collaborative manner without the need to share sensitive data.<n>It remains vulnerable to Gradient Leakage Attacks (GLAs), which can reveal private information from the shared model updates.<n>We investigate the effectiveness of Differential Privacy mechanisms as defenses against GLAs.
  arXiv  Detail & Related papers  (2025-10-27T23:33:21Z)
• The Hidden Cost of Modeling P(X): Vulnerability to Membership Inference Attacks in Generative Text Classifiers [6.542294761666199]
  Membership Inference Attacks (MIAs) pose a critical privacy threat by enabling adversaries to determine whether a specific sample was included in a model's training dataset.<n>We show that fully generative classifiers which explicitly model the joint likelihood $P(X,Y)$ are most vulnerable to membership leakage.
  arXiv  Detail & Related papers  (2025-10-17T18:09:33Z)
• On the MIA Vulnerability Gap Between Private GANs and Diffusion Models [51.53790101362898]
  Generative Adversarial Networks (GANs) and diffusion models have emerged as leading approaches for high-quality image synthesis.<n>We present the first unified theoretical and empirical analysis of the privacy risks faced by differentially private generative models.
  arXiv  Detail & Related papers  (2025-09-03T14:18:22Z)
• Evaluating the Dynamics of Membership Privacy in Deep Learning [9.899573367308506]
  Membership inference attacks (MIAs) pose a critical threat to the privacy of training data in deep learning.<n>This paper presents a dynamic analytical framework for dissecting and quantifying privacy leakage dynamics at the individual sample level.
  arXiv  Detail & Related papers  (2025-07-31T07:09:52Z)
• When Better Features Mean Greater Risks: The Performance-Privacy Trade-Off in Contrastive Learning [9.660010886245155]
  This paper systematically investigates the privacy threats posed by membership inference attacks (MIAs) targeting encoder models.<n>We propose a novel membership inference attack method based on the p-norm of feature vectors, termed the Embedding Lp-Norm Likelihood Attack (LpLA)
  arXiv  Detail & Related papers  (2025-06-06T05:03:29Z)
• Unveiling the Unseen: Exploring Whitebox Membership Inference through the Lens of Explainability [10.632831321114502]
  We propose an attack-driven explainable framework to identify the most influential features of raw data that lead to successful membership inference attacks. Our proposed MIA shows an improvement of up to 26% on state-of-the-art MIA.
  arXiv  Detail & Related papers  (2024-07-01T14:07:46Z)
• Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift [104.76588209308666]
  This paper explores backdoor attacks in LVLM instruction tuning across mismatched training and testing domains.<n>We introduce a new evaluation dimension, backdoor domain generalization, to assess attack robustness.<n>We propose a multimodal attribution backdoor attack (MABA) that injects domain-agnostic triggers into critical areas.
  arXiv  Detail & Related papers  (2024-06-27T02:31:03Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models [64.03398193325572]
  Inference attacks against Machine Learning (ML) models allow adversaries to learn about training data, model parameters, etc. We concentrate on four attacks - namely, membership inference, model inversion, attribute inference, and model stealing. Our analysis relies on a modular re-usable software, ML-Doctor, which enables ML model owners to assess the risks of deploying their models.
  arXiv  Detail & Related papers  (2021-02-04T11:35:13Z)
• Curse or Redemption? How Data Heterogeneity Affects the Robustness of Federated Learning [51.15273664903583]
  Data heterogeneity has been identified as one of the key features in federated learning but often overlooked in the lens of robustness to adversarial attacks. This paper focuses on characterizing and understanding its impact on backdooring attacks in federated learning through comprehensive experiments using synthetic and the LEAF benchmarks.
  arXiv  Detail & Related papers  (2021-02-01T06:06:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.