Primal-Dual Direct Preference Optimization for Constrained LLM Alignment

• URL: http://arxiv.org/abs/2510.05703v1
• Date: Tue, 07 Oct 2025 09:10:35 GMT
• Title: Primal-Dual Direct Preference Optimization for Constrained LLM Alignment
• Abstract summary: We study the problem of constrained alignment in Large Language Models (LLMs)<n>We propose a novel primal-dual DPO approach, which first trains a model using standard DPO on reward preference data.<n>We establish rigorous theoretical guarantees on the suboptimality and constraint violation of the output policy.
• Score: 16.080857375857697
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The widespread application of Large Language Models (LLMs) imposes increasing demands on safety, such as reducing harmful content and fake information, and avoiding certain forbidden tokens due to rules and laws. While there have been several recent works studying safe alignment of LLMs, these works either require the training of reward and cost models and incur high memory and computational costs, or need prior knowledge about the optimal solution. Motivated by this fact, we study the problem of constrained alignment in LLMs, i.e., maximizing the output reward while restricting the cost due to potentially unsafe content to stay below a threshold. For this problem, we propose a novel primal-dual DPO approach, which first trains a model using standard DPO on reward preference data to provide reward information, and then adopts a rearranged Lagrangian DPO objective utilizing the provided reward information to fine-tune LLMs on cost preference data. Our approach significantly reduces memory and computational costs, and does not require extra prior knowledge. Moreover, we establish rigorous theoretical guarantees on the suboptimality and constraint violation of the output policy. We also extend our approach to an online data setting by incorporating exploration bonuses, which enables our approach to explore uncovered prompt-response space, and then provide theoretical results that get rid of the dependence on preference data coverage. Experimental results on the widely-used preference dataset PKU-SafeRLHF demonstrate the effectiveness of our approach.

Related papers

• PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment [10.476492988819372]
  We introduce PREFINE: Preference-based Implicit Reward and Cost Fine-Tuning for Safety Alignment.<n> PREFINE produces policies that achieve low-cost, high-reward performance with significantly improved data and computational efficiency.
  arXiv  Detail & Related papers  (2026-05-20T14:19:45Z)
• Data-dependent Exploration for Online Reinforcement Learning from Human Feedback [50.34161049551627]
  Online reinforcement learning from human feedback (RLHF) has emerged as a promising paradigm for aligning large language models (LLMs) by continuously collecting new preference feedback during training.<n>Existing exploration strategies often derive bonuses via on-policy expectations, which are difficult to estimate reliably from the limited historical preference data available during training.<n>We propose data-dependent exploration for preference optimization (DEPO), a simple and scalable method that leverages historical data to construct an extra uncertainty bonus for high-uncertainty regions.
  arXiv  Detail & Related papers  (2026-05-06T03:56:45Z)
• Safe Reinforcement Learning with Preference-based Constraint Inference [28.363990267209584]
  We propose Preference-based Constrained Reinforcement Learning (PbCRL) for constraint inference in Safe RL.<n>PbCRL achieves superior alignment with true safety requirements and outperforms the state-of-the-art baselines in terms of safety and reward.
  arXiv  Detail & Related papers  (2026-03-24T08:54:03Z)
• Towards Sample-Efficient and Stable Reinforcement Learning for LLM-based Recommendation [56.92367609590823]
  Long Chain-of-Thought (Long CoT) reasoning has shown promise in Large Language Models (LLMs)<n>We argue that Long CoT is inherently ill-suited for the sequential recommendation domain.<n>We propose RISER, a novel Reinforced Item Space Exploration framework for Recommendation.
  arXiv  Detail & Related papers  (2026-01-31T10:02:43Z)
• Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking [78.69179041551014]
  We propose an information-theoretic reward modeling framework based on the Information Bottleneck principle.<n>We show that InfoRM filters out preference-irrelevant information to alleviate reward misgeneralization.<n>We also introduce IBL, a distribution-level regularization that penalizes such deviations, effectively expanding the optimization landscape.
  arXiv  Detail & Related papers  (2025-10-15T15:51:59Z)
• Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning [26.120338506874976]
  Unlearning, which aims to remove the influence of specific data while preserving overall model utility, is becoming an important research area.<n>We derive a novel unlearning algorithm termed textbfDistribution textbfPreference textbfOptimization (DiPO)<n>DiPO attains the highest forget quality on the TOFU benchmark, and maintains leading scalability and sustainability on the MUSE benchmark.
  arXiv  Detail & Related papers  (2025-10-06T12:49:00Z)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression [52.0792918455501]
  We propose a novel two-stage policy optimization framework that directly approximates the optimal advantage function.<n>$A$*-PO achieves competitive performance across a wide range of mathematical reasoning benchmarks.<n>It reduces training time by up to 2$times$ and peak memory usage by over 30% compared to PPO, GRPO, and REBEL.
  arXiv  Detail & Related papers  (2025-05-27T03:58:50Z)
• Large Language Model driven Policy Exploration for Recommender Systems [50.70228564385797]
  offline RL policies trained on static user data are vulnerable to distribution shift when deployed in dynamic online environments.<n>Online RL-based RS also face challenges in production deployment due to the risks of exposing users to untrained or unstable policies.<n>Large Language Models (LLMs) offer a promising solution to mimic user objectives and preferences for pre-training policies offline.<n>We propose an Interaction-Augmented Learned Policy (iALP) that utilizes user preferences distilled from an LLM.
  arXiv  Detail & Related papers  (2025-01-23T16:37:44Z)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs [63.32585910975191]
  We introduce reward-conditioned Large Language Models (LLMs) that learn from the entire spectrum of response quality within the dataset.<n>We show that our approach consistently boosts DPO by a considerable margin.<n>Our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere data expansion.
  arXiv  Detail & Related papers  (2024-10-10T16:01:51Z)
• Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment [7.577032515701217]
  We investigate the vulnerability of Large Language Models aligned using two methods - DPO and PPO - to membership inference attacks (MIAs)<n>We introduce a novel reference-based attack framework specifically for analyzing preference data called PREMIA (ulinePreference data ulineMIA)
  arXiv  Detail & Related papers  (2024-07-08T22:53:23Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion [8.499811428928071]
  We propose the Budget Constrained Reinforcement Learning for Sequential Promotion framework to determine the value of cash bonuses to be sent to users. We show that BCRLSP achieves a higher long-term customer retention rate and a lower cost than various baselines.
  arXiv  Detail & Related papers  (2022-07-16T00:10:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.