Suan: Rectifying Direct Preference Safety Alignment in Large Language Models

• URL: http://arxiv.org/abs/2609.08634v1
• Date: Tue, 08 Sep 2026 12:04:38 GMT
• Title: Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
• Abstract summary: We introduce Suan, a novel preference optimization algorithm.<n>We obtain more interpretable and robust training dynamics.<n>In evaluations, Suan achieves superior safety alignment while fully preserving response utility.
• Score: 15.119045051735633
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility.

Related papers

• Selective Safety Steering via Value-Filtered Decoding [54.87935112120107]
  Large language models (LLMs) are trained to align with human values, but their generations may still violate safety constraints.<n>Existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model.<n>We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of unsafe responses.
  arXiv  Detail & Related papers  (2026-05-14T12:13:08Z)
• BSO: Safety Alignment Is Density Ratio Matching [14.960171907231663]
  We show that the likelihood of the optimal safe policy admits a closed-form decomposition that reduces safety alignment to a density ratio matching problem.<n>Minimizing Bregman divergences between the data and model ratios yields Bregman Safety Optimization (BSO), a family of single-stage loss functions, that provably recover the optimal safe policy.<n> Experiments across safety alignment benchmarks show that BSO consistently improves the safety-helpfulness trade-off.
  arXiv  Detail & Related papers  (2026-05-12T16:19:42Z)
• Improving Safety Alignment via Balanced Direct Preference Optimization [36.066367724362614]
  Reinforcement Learning from Human Feedback (RLHF) has been adopted to enhance the safety performance of Large Language Models (LLMs)<n>Direct Preference Optimization (DPO) is widely used for safety alignment.<n>This paper revisits the overfitting phenomenon from the perspective of the model's comprehension of the training data.
  arXiv  Detail & Related papers  (2026-03-24T06:04:09Z)
• Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models [57.006252510102506]
  Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications.<n>We introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems.
  arXiv  Detail & Related papers  (2026-02-12T22:03:35Z)
• IntentionReasoner: Facilitating Adaptive LLM Safeguards through Intent Reasoning and Selective Query Refinement [35.904652937034136]
  We introduce IntentionReasoner, a novel safeguard mechanism that leverages a dedicated guard model to perform intent reasoning.<n>We show that IntentionReasoner excels in multiple safeguard benchmarks, generation quality evaluations, and jailbreak attack scenarios.
  arXiv  Detail & Related papers  (2025-08-27T16:47:31Z)
• Rethinking Safety in LLM Fine-tuning: An Optimization Perspective [56.31306558218838]
  We show that poor optimization choices, rather than inherent trade-offs, often cause safety problems, measured as harmful responses to adversarial prompts.<n>We propose a simple exponential moving average (EMA) momentum technique in parameter space that preserves safety performance.<n>Our experiments on the Llama families across multiple datasets demonstrate that safety problems can largely be avoided without specialized interventions.
  arXiv  Detail & Related papers  (2025-08-17T23:46:36Z)
• SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety [57.14003339251827]
  We introduce a new algorithm called SafeDPO, which is designed to directly optimize the safety alignment objective in a single stage of policy learning.<n>As a result, it eliminates the need to fit separate reward and cost models or to sample from the language model during fine-tuning.<n>We demonstrate that SafeDPO achieves competitive performance compared to state-of-the-art safety alignment algorithms.
  arXiv  Detail & Related papers  (2025-05-26T14:50:01Z)
• Multi-head Reward Aggregation Guided by Entropy [1.7505359879440667]
  Rules characterized by high rating entropy are less reliable in identifying responses preferred by humans.<n>We introduce ENCORE, a straightforward entropy-guided approach that composes multi-head rewards by downweighting rules exhibiting high rating entropy.<n>Our proposed approach is training-free, broadly applicable to various datasets, and maintains interpretability.
  arXiv  Detail & Related papers  (2025-03-26T21:16:48Z)
• Deliberative Alignment: Reasoning Enables Safer Language Models [64.60765108418062]
  We introduce Deliberative Alignment, a new paradigm that teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering.<n>We used this approach to align OpenAI's o-series models, and achieved highly precise adherence to OpenAI's safety policies, without requiring human-written chain-of-thoughts or answers.
  arXiv  Detail & Related papers  (2024-12-20T21:00:11Z)
• Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training [67.30423823744506]
  We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position.<n>DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence.
  arXiv  Detail & Related papers  (2024-07-12T09:36:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.