PROPS: Progressively Private Self-alignment of Large Language Models

• URL: http://arxiv.org/abs/2508.06783v1
• Date: Sat, 09 Aug 2025 02:17:47 GMT
• Title: PROPS: Progressively Private Self-alignment of Large Language Models
• Abstract summary: This work focuses on Large Language Models (LLMs) alignment with preference-level privacy.<n>We propose PROPS (PROgressively Private Self-alignment), a multi-stage privacy preserving alignment framework.<n>For the same privacy budget, alignment via PROPS can achieve up to 3x higher win-rates compared to DP-SGD, and 2.5x higher win-rates compared to Randomized Response (RR) based alignment.
• Score: 16.049154157152177
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Alignment is a key step in developing Large Language Models (LLMs) using human feedback to ensure adherence to human values and societal norms. Dependence on human feedback raises privacy concerns about how much a labeler's preferences may reveal about their personal values, beliefs, and personality traits. Existing approaches, such as Differentially Private SGD (DP-SGD), provide rigorous privacy guarantees by privatizing gradients during fine-tuning and alignment but can provide more privacy than necessary as human preferences are tied only to labels of (prompt, response) pairs and can degrade model utility. This work focuses on LLM alignment with preference-level privacy, which preserves the privacy of preference labels provided by humans. We propose PROPS (PROgressively Private Self-alignment), a multi-stage privacy preserving alignment framework where privately aligned models in previous stages can serve as labelers for supplementing training data in the subsequent stages of alignment. We present theoretical guarantees for PROPS as well as comprehensive validation using multiple models (Pythia and GPT) and datasets (AlpacaEval, Anthropic HH-RLHF, truthy-dpo-v0.1) to demonstrate the utility of PROPS over existing methods while still providing high privacy. For the same privacy budget, alignment via PROPS can achieve up to 3x higher win-rates compared to DP-SGD, and 2.5x higher win-rates compared to Randomized Response (RR) based alignment.

Related papers

• Privacy Without Regret: Differentially Private Inference-Time Alignment [8.568849346613602]
  We show that a single intervention-adding noise to reward scores before selection-resolves both.<n>PrivBoN establishes that Gumbel noise at an appropriate scale simultaneously provides $$-differential privacy.<n>PrivITP achieves ex-post $(,)$-DP with a privacy cost independent of the number of responses.
  arXiv  Detail & Related papers  (2026-08-26T18:56:27Z)
• Private Direct Preference Optimization for LLM Alignment [77.56374641532715]
  Direct preference optimization (DPO) is now a standard method for aligning large language models (LLMs) using human preference data.<n>In this paper, we formalize preference privacy, a label-DP-style privacy notion for DPO.<n>We then design PrivDPO, a DPO variant that enforces preference privacy while remaining compatible with large-scale LLM training.
  arXiv  Detail & Related papers  (2026-08-05T16:51:02Z)
• Machine Learning with Privacy for Protected Attributes [56.44253915927481]
  We refine the definition of differential privacy (DP) to create a more general and flexible framework that we call feature differential privacy (FDP)<n>Our definition is simulation-based and allows for both addition/removal and replacement variants of privacy, and can handle arbitrary separation of protected and non-protected features.<n>We apply our framework to various machine learning tasks and show that it can significantly improve the utility of DP-trained models when public features are available.
  arXiv  Detail & Related papers  (2025-06-24T17:53:28Z)
• Improved Algorithms for Differentially Private Language Model Alignment [12.611907955831597]
  We propose novel algorithms for privacy-preserving alignment.<n>Our framework can be deployed on two celebrated alignment techniques.<n>One of our algorithms, DP-AdamW, combined with DPO, surpasses existing methods, improving alignment quality by up to 15% under moderate privacy budgets.
  arXiv  Detail & Related papers  (2025-05-13T16:18:59Z)
• Differentially Private 2D Human Pose Estimation [6.982542225631412]
  We present the first comprehensive framework for differentially private 2D human pose estimation (2D-HPE)<n>To effectively balance privacy performance, we adopt Projected DP-SGD, which projects the noisy gradients to a low-dimensional subspace.<n>Next, we incorporate Feature Differential Privacy(FDP) to selectively privatize only sensitive features while retaining public visual cues.
  arXiv  Detail & Related papers  (2025-04-14T12:50:37Z)
• Enhancing Feature-Specific Data Protection via Bayesian Coordinate Differential Privacy [55.357715095623554]
  Local Differential Privacy (LDP) offers strong privacy guarantees without requiring users to trust external parties. We propose a Bayesian framework, Bayesian Coordinate Differential Privacy (BCDP), that enables feature-specific privacy quantification.
  arXiv  Detail & Related papers  (2024-10-24T03:39:55Z)
• Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning [62.224804688233]
  differential privacy (DP) offers a promising solution by ensuring models are 'almost indistinguishable' with or without any particular privacy unit. We study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users.
  arXiv  Detail & Related papers  (2024-06-20T13:54:32Z)
• Personalized Differential Privacy for Ridge Regression [3.4751583941317166]
  We introduce our novel Personalized-DP Output Perturbation method ( PDP-OP) that enables to train Ridge regression models with individual per data point privacy levels. We provide rigorous privacy proofs for our PDP-OP as well as accuracy guarantees for the resulting model. We show that PDP-OP outperforms the personalized privacy techniques of Jorgensen et al.
  arXiv  Detail & Related papers  (2024-01-30T16:00:14Z)
• Private Fine-tuning of Large Language Models with Zeroth-order Optimization [51.19403058739522]
  Differentially private gradient descent (DP-SGD) allows models to be trained in a privacy-preserving manner.<n>We introduce DP-ZO, a private fine-tuning framework for large language models by privatizing zeroth order optimization methods.
  arXiv  Detail & Related papers  (2024-01-09T03:53:59Z)
• PAC Privacy Preserving Diffusion Models [6.299952353968428]
  Diffusion models can produce images with both high privacy and visual quality.<n>However, challenges arise such as in ensuring robust protection in privatizing specific data attributes.<n>We introduce the PAC Privacy Preserving Diffusion Model, a model leverages diffusion principles and ensure Probably Approximately Correct (PAC) privacy.
  arXiv  Detail & Related papers  (2023-12-02T18:42:52Z)
• Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent [69.14164921515949]
  We characterize privacy guarantees for individual examples when releasing models trained by DP-SGD. We find that most examples enjoy stronger privacy guarantees than the worst-case bound. This implies groups that are underserved in terms of model utility simultaneously experience weaker privacy guarantees.
  arXiv  Detail & Related papers  (2022-06-06T13:49:37Z)
• Just Fine-tune Twice: Selective Differential Privacy for Large Language Models [69.66654761324702]
  We propose a simple yet effective just-fine-tune-twice privacy mechanism to achieve SDP for large Transformer-based language models. Experiments show that our models achieve strong performance while staying robust to the canary insertion attack.
  arXiv  Detail & Related papers  (2022-04-15T22:36:55Z)
• Personalized PATE: Differential Privacy for Machine Learning with Individual Privacy Guarantees [1.2691047660244335]
  We propose three novel methods to support training an ML model with different personalized privacy guarantees within the training data. Our experiments show that our personalized privacy methods yield higher accuracy models than the non-personalized baseline.
  arXiv  Detail & Related papers  (2022-02-21T20:16:27Z)
• Private Reinforcement Learning with PAC and Regret Guarantees [69.4202374491817]
  We design privacy preserving exploration policies for episodic reinforcement learning (RL) We first provide a meaningful privacy formulation using the notion of joint differential privacy (JDP) We then develop a private optimism-based learning algorithm that simultaneously achieves strong PAC and regret bounds, and enjoys a JDP guarantee.
  arXiv  Detail & Related papers  (2020-09-18T20:18:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.