BSO: Safety Alignment Is Density Ratio Matching

• URL: http://arxiv.org/abs/2605.12339v1
• Date: Tue, 12 May 2026 16:19:42 GMT
• Title: BSO: Safety Alignment Is Density Ratio Matching
• Abstract summary: We show that the likelihood of the optimal safe policy admits a closed-form decomposition that reduces safety alignment to a density ratio matching problem.<n>Minimizing Bregman divergences between the data and model ratios yields Bregman Safety Optimization (BSO), a family of single-stage loss functions, that provably recover the optimal safe policy.<n> Experiments across safety alignment benchmarks show that BSO consistently improves the safety-helpfulness trade-off.
• Score: 14.960171907231663
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Aligning language models for both helpfulness and safety typically requires complex pipelines-separate reward and cost models, online reinforcement learning, and primal-dual updates. Recent direct preference optimization approaches simplify training but incorporate safety through ad-hoc modifications such as multi-stage procedures or heuristic margin terms, lacking a principled derivation. We show that the likelihood ratio of the optimal safe policy admits a closed-form decomposition that reduces safety alignment to a density ratio matching problem. Minimizing Bregman divergences between the data and model ratios yields Bregman Safety Optimization (BSO), a family of single-stage loss functions, each induced by a convex generator, that provably recover the optimal safe policy. BSO is both general and simple: it requires no auxiliary models, introduces only one hyperparameter beyond standard preference optimization, and recovers existing safety-aware methods as special cases. Experiments across safety alignment benchmarks show that BSO consistently improves the safety-helpfulness trade-off.

Related papers

• Suan: Rectifying Direct Preference Safety Alignment in Large Language Models [15.119045051735633]
  We introduce Suan, a novel preference optimization algorithm.<n>We obtain more interpretable and robust training dynamics.<n>In evaluations, Suan achieves superior safety alignment while fully preserving response utility.
  arXiv  Detail & Related papers  (2026-09-08T12:04:38Z)
• Configurable Reward Model for Balanced Safety Alignment [48.459540031156955]
  Existing instruction-tuned large language models (LLMs) often fail to generalize to new safety configurations.<n>We introduce the Safety Reward Model (CSRM), which is jointly optimized for safety compliance and reward modeling.<n>The resulting RM is sensitive to fine-grained safety configurations and conversational nuances, substantially improving generalization to previously unseen safety configurations.
  arXiv  Detail & Related papers  (2026-05-28T19:05:53Z)
• Selective Safety Steering via Value-Filtered Decoding [54.87935112120107]
  Large language models (LLMs) are trained to align with human values, but their generations may still violate safety constraints.<n>Existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model.<n>We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of unsafe responses.
  arXiv  Detail & Related papers  (2026-05-14T12:13:08Z)
• Improving Safety Alignment via Balanced Direct Preference Optimization [36.066367724362614]
  Reinforcement Learning from Human Feedback (RLHF) has been adopted to enhance the safety performance of Large Language Models (LLMs)<n>Direct Preference Optimization (DPO) is widely used for safety alignment.<n>This paper revisits the overfitting phenomenon from the perspective of the model's comprehension of the training data.
  arXiv  Detail & Related papers  (2026-03-24T06:04:09Z)
• SafeR-CLIP: Mitigating NSFW Content in Vision-Language Models While Preserving Pre-Trained Knowledge [51.634837361795434]
  SaFeR-CLIP reconciles safety and performance, recovering up to 8.0% in zero-shot accuracy over prior methods.<n>We also contribute NSFW-Caps, a new benchmark of 1,000 highly-aligned pairs for testing safety under distributional shift.
  arXiv  Detail & Related papers  (2025-11-20T19:00:15Z)
• Rethinking Safety in LLM Fine-tuning: An Optimization Perspective [56.31306558218838]
  We show that poor optimization choices, rather than inherent trade-offs, often cause safety problems, measured as harmful responses to adversarial prompts.<n>We propose a simple exponential moving average (EMA) momentum technique in parameter space that preserves safety performance.<n>Our experiments on the Llama families across multiple datasets demonstrate that safety problems can largely be avoided without specialized interventions.
  arXiv  Detail & Related papers  (2025-08-17T23:46:36Z)
• SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety [57.14003339251827]
  We introduce a new algorithm called SafeDPO, which is designed to directly optimize the safety alignment objective in a single stage of policy learning.<n>As a result, it eliminates the need to fit separate reward and cost models or to sample from the language model during fine-tuning.<n>We demonstrate that SafeDPO achieves competitive performance compared to state-of-the-art safety alignment algorithms.
  arXiv  Detail & Related papers  (2025-05-26T14:50:01Z)
• LookAhead Tuning: Safer Language Models via Partial Answer Previews [62.529794567687354]
  Fine-tuning enables large language models to adapt to specific domains, but often compromises their previously established safety alignment.<n>We introduce LookAhead Tuning, a lightweight and effective data-driven approach that preserves safety during fine-tuning.
  arXiv  Detail & Related papers  (2025-03-24T18:11:42Z)
• Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements [46.79887158348167]
  The current paradigm for safety alignment of large language models (LLMs) follows a one-size-fits-all approach.<n>We propose Controllable Safety Alignment (CoSA), a framework designed to adapt models to diverse safety requirements without re-training.
  arXiv  Detail & Related papers  (2024-10-11T16:38:01Z)
• One-Shot Safety Alignment for Large Language Models via Optimal Dualization [64.52223677468861]
  This paper presents a perspective of dualization that reduces constrained alignment to an equivalent unconstrained alignment problem. We do so by pre-optimizing a smooth and convex dual function that has a closed form. Our strategy leads to two practical algorithms in model-based and preference-based settings.
  arXiv  Detail & Related papers  (2024-05-29T22:12:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.