Improving Safety Alignment via Balanced Direct Preference Optimization

• URL: http://arxiv.org/abs/2603.22829v1
• Date: Tue, 24 Mar 2026 06:04:09 GMT
• Title: Improving Safety Alignment via Balanced Direct Preference Optimization
• Abstract summary: Reinforcement Learning from Human Feedback (RLHF) has been adopted to enhance the safety performance of Large Language Models (LLMs)<n>Direct Preference Optimization (DPO) is widely used for safety alignment.<n>This paper revisits the overfitting phenomenon from the perspective of the model's comprehension of the training data.
• Score: 36.066367724362614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid development and widespread application of Large Language Models (LLMs), their potential safety risks have attracted widespread attention. Reinforcement Learning from Human Feedback (RLHF) has been adopted to enhance the safety performance of LLMs. As a simple and effective alternative to RLHF, Direct Preference Optimization (DPO) is widely used for safety alignment. However, safety alignment still suffers from severe overfitting, which limits its actual performance. This paper revisits the overfitting phenomenon from the perspective of the model's comprehension of the training data. We find that the Imbalanced Preference Comprehension phenomenon exists between responses in preference pairs, which compromises the model's safety performance. To address this, we propose Balanced Direct Preference Optimization (B-DPO), which adaptively modulates optimization strength between preferred and dispreferred responses based on mutual information. A series of experimental results show that B-DPO can enhance the safety capability while maintaining the competitive general capabilities of LLMs on various mainstream benchmarks compared to state-of-the-art methods. \color{red}{Warning: This paper contains examples of harmful texts, and reader discretion is recommended.

Related papers

• Suan: Rectifying Direct Preference Safety Alignment in Large Language Models [15.119045051735633]
  We introduce Suan, a novel preference optimization algorithm.<n>We obtain more interpretable and robust training dynamics.<n>In evaluations, Suan achieves superior safety alignment while fully preserving response utility.
  arXiv  Detail & Related papers  (2026-09-08T12:04:38Z)
• BSO: Safety Alignment Is Density Ratio Matching [14.960171907231663]
  We show that the likelihood of the optimal safe policy admits a closed-form decomposition that reduces safety alignment to a density ratio matching problem.<n>Minimizing Bregman divergences between the data and model ratios yields Bregman Safety Optimization (BSO), a family of single-stage loss functions, that provably recover the optimal safe policy.<n> Experiments across safety alignment benchmarks show that BSO consistently improves the safety-helpfulness trade-off.
  arXiv  Detail & Related papers  (2026-05-12T16:19:42Z)
• Rethinking Safety in LLM Fine-tuning: An Optimization Perspective [56.31306558218838]
  We show that poor optimization choices, rather than inherent trade-offs, often cause safety problems, measured as harmful responses to adversarial prompts.<n>We propose a simple exponential moving average (EMA) momentum technique in parameter space that preserves safety performance.<n>Our experiments on the Llama families across multiple datasets demonstrate that safety problems can largely be avoided without specialized interventions.
  arXiv  Detail & Related papers  (2025-08-17T23:46:36Z)
• SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety [57.14003339251827]
  We introduce a new algorithm called SafeDPO, which is designed to directly optimize the safety alignment objective in a single stage of policy learning.<n>As a result, it eliminates the need to fit separate reward and cost models or to sample from the language model during fine-tuning.<n>We demonstrate that SafeDPO achieves competitive performance compared to state-of-the-art safety alignment algorithms.
  arXiv  Detail & Related papers  (2025-05-26T14:50:01Z)
• SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning [51.78514648677898]
  We propose SAFER, a framework for Safety Alignment via eFficient Ex-Ante Reasoning.<n>Our approach instantiates structured Ex-Ante reasoning through initial assessment, rule verification, and path calibration.<n> Experiments on multiple open-source LLMs demonstrate that SAFER significantly enhances safety performance while maintaining helpfulness and response efficiency.
  arXiv  Detail & Related papers  (2025-04-03T16:07:38Z)
• LookAhead Tuning: Safer Language Models via Partial Answer Previews [62.529794567687354]
  Fine-tuning enables large language models to adapt to specific domains, but often compromises their previously established safety alignment.<n>We introduce LookAhead Tuning, a lightweight and effective data-driven approach that preserves safety during fine-tuning.
  arXiv  Detail & Related papers  (2025-03-24T18:11:42Z)
• Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling [84.00480999255628]
  Reinforcement Learning algorithms for safety alignment of Large Language Models (LLMs) encounter the challenge of distribution shift.<n>Current approaches typically address this issue through online sampling from the target policy.<n>We propose a new framework that leverages the model's intrinsic safety judgment capability to extract reward signals.
  arXiv  Detail & Related papers  (2025-03-13T06:40:34Z)
• Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models [24.168387024091082]
  Fine-tuning large language models (LLMs) based on human preferences has been effective in improving their performance.<n>Maintaining safety throughout the fine-tuning process remains a significant challenge.<n>We propose an Equilibrate RLHF framework that achieves better safety alignment even with fewer training data.
  arXiv  Detail & Related papers  (2025-02-17T08:40:30Z)
• Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models [94.39278422567955]
  Fine-tuning large language models (LLMs) on human preferences has proven successful in enhancing their capabilities.<n>However, ensuring the safety of LLMs during fine-tuning remains a critical concern.<n>We propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO) to address this issue.
  arXiv  Detail & Related papers  (2024-08-27T17:31:21Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Enhancing LLM Safety via Constrained Direct Preference Optimization [8.22888921018027]
  We introduce Constrained DPO (C-DPO), a novel extension of the recently proposed Direct Preference Optimization (DPO) approach for fine-tuning AI systems. By integrating dual gradient descent and DPO, our method identifies a nearly optimal trade-off between helpfulness and harmlessness without using reinforcement learning. Empirically, our approach provides a safety guarantee to LLMs that is missing in DPO while achieving significantly higher rewards under the same safety constraint.
  arXiv  Detail & Related papers  (2024-03-04T20:39:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.