Private Direct Preference Optimization for LLM Alignment

• URL: http://arxiv.org/abs/2608.05040v1
• Date: Wed, 05 Aug 2026 16:51:02 GMT
• Title: Private Direct Preference Optimization for LLM Alignment
• Abstract summary: Direct preference optimization (DPO) is now a standard method for aligning large language models (LLMs) using human preference data.<n>In this paper, we formalize preference privacy, a label-DP-style privacy notion for DPO.<n>We then design PrivDPO, a DPO variant that enforces preference privacy while remaining compatible with large-scale LLM training.
• Score: 77.56374641532715
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Direct preference optimization (DPO) is now a standard method for aligning large language models (LLMs) using human preference data. Each DPO example contains a prompt and a pair of candidate model responses. While prompts and responses are often public or model-generated, the relative preference between responses reflects subjective judgments and can reveal sensitive attributes of annotators or end users. Off-the-shelf privacy-preserving approaches are not well matched to this structure, leading to unnecessary noise injection and biased updates in training. In this paper, we formalize preference privacy, a label-DP-style privacy notion for DPO that protects only the relative preference between candidate responses, assuming an adversary who already knows the prompt and responses. We then design PrivDPO, a DPO variant that enforces preference privacy while remaining compatible with large-scale LLM training. Our main observation is that, for neighboring examples differing only in their preference signal, the gradient difference lies on a one-dimensional preference axis determined solely by the text; all preference information flows through this axis. PrivDPO adds calibrated randomness only along this axis via an unbiased randomized rescaling of the DPO objective, avoiding per-example gradient operations. Our experiments on three alignment benchmarks and three LLM families show that PrivDPO consistently achieves strong privacy-utility trade-offs compared with privacy-preserving baselines.

Related papers

• Mult-DPO: Multinomial Direct Preference Optimization for Recommender Systems [72.27018654934386]
  Direct preference optimization (DPO) is a simple and effective alignment strategy for large language models (LLMs) based on pairwise preferences.<n>In recommender systems, however, user feedback is rarely pairwise.<n>We propose Mult-DPO, a tractable multinomial surrogate likelihood over set-wise preference events for the user-preference alignment of LLM-based recommender systems.
  arXiv  Detail & Related papers  (2026-06-08T18:53:45Z)
• Differentially Private Preference Data Synthesis for Large Language Model Alignment [6.989255131957306]
  DPPrefSyn is an algorithm for generating differentially private (DP) synthetic preference data.<n>It exploits the shared linear structure of per-cluster reward models to capture heterogeneous human preferences in private datasets.
  arXiv  Detail & Related papers  (2026-05-29T03:53:12Z)
• wDPO: Winsorized Direct Preference Optimization for Robust LLM Alignment [48.487557157323664]
  In practice, preference data are often noisy.<n>Existing robust variants of DPO mainly rely on uniform objective modifications or global reweighting.<n>We show that robust preference alignment benefits from addressing different noise types with targeted interventions.
  arXiv  Detail & Related papers  (2026-03-07T13:30:53Z)
• Robust Preference Alignment via Directional Neighborhood Consensus [13.313830197011983]
  We introduce Robust Preference Selection (RPS), a post-hoc, training-free method by leveraging directional neighborhood consensus.<n>RPS samples multiple responses from a local neighborhood of related preferences to create a superior candidate pool.<n>Our work presents a practical, theoretically-grounded solution for enhancing the reliability of preference-aligned models.
  arXiv  Detail & Related papers  (2025-10-23T12:39:20Z)
• On the Role of Preference Variance in Preference Optimization [55.364953481473286]
  We investigate the impact of emphpreference variance (PVar) on the effectiveness of Direct Preference Optimization (DPO) training.<n>We show that prompts with higher PVar outperform randomly selected prompts or those with lower PVar.
  arXiv  Detail & Related papers  (2025-10-14T22:34:52Z)
• PROPS: Progressively Private Self-alignment of Large Language Models [16.049154157152177]
  This work focuses on Large Language Models (LLMs) alignment with preference-level privacy.<n>We propose PROPS (PROgressively Private Self-alignment), a multi-stage privacy preserving alignment framework.<n>For the same privacy budget, alignment via PROPS can achieve up to 3x higher win-rates compared to DP-SGD, and 2.5x higher win-rates compared to Randomized Response (RR) based alignment.
  arXiv  Detail & Related papers  (2025-08-09T02:17:47Z)
• Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment [7.577032515701217]
  We investigate the vulnerability of Large Language Models aligned using two methods - DPO and PPO - to membership inference attacks (MIAs)<n>We introduce a novel reference-based attack framework specifically for analyzing preference data called PREMIA (ulinePreference data ulineMIA)
  arXiv  Detail & Related papers  (2024-07-08T22:53:23Z)
• Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment [72.99676237703099]
  We propose a new framework that boosts the alignment of large language models with human preferences.<n>Our key idea is leveraging the human prior knowledge within the small (seed) data.<n>We introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data.
  arXiv  Detail & Related papers  (2024-06-06T18:01:02Z)
• Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization [105.3612692153615]
  We propose a new axis based on eliciting preferences jointly over instruction-response pairs.<n>Joint preferences over instruction and response pairs can significantly enhance the alignment of large language models.
  arXiv  Detail & Related papers  (2024-03-31T02:05:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.