Related papers: Configurable Reward Model for Balanced Safety Alignment

Configurable Reward Model for Balanced Safety Alignment

URL: http://arxiv.org/abs/2605.30487v1
Date: Thu, 28 May 2026 19:05:53 GMT
Title: Configurable Reward Model for Balanced Safety Alignment
Abstract summary: Existing instruction-tuned large language models (LLMs) often fail to generalize to new safety configurations.<n>We introduce the Safety Reward Model (CSRM), which is jointly optimized for safety compliance and reward modeling.<n>The resulting RM is sensitive to fine-grained safety configurations and conversational nuances, substantially improving generalization to previously unseen safety configurations.
Score: 48.459540031156955
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Aligning large language models (LLMs) to heterogeneous and rapidly evolving safety requirements remains a critical challenge. Existing instruction-tuned LLMs and standalone safety classifiers often fail to generalize to new safety configurations, motivating the need for Reward Models (RMs) that are explicitly configurable to changing specifications. We introduce the Configurable Safety Reward Model (CSRM), which is jointly optimized for calibrated safety compliance and reward modeling. Our approach is supported by configuration-targeted data augmentation that enforces instruction adherence while preserving relative severity structure. The resulting RM is sensitive to fine-grained safety configurations and conversational nuances, substantially improving generalization to previously unseen safety configurations. CSRM achieves state-of-the-art performance on recent configurable safety benchmarks, including CoSApien (94.6% F1) and DynaBench (75.8% F1), without requiring additional human annotation. When used for downstream safety alignment, CSRM yields LLMs with a significantly improved helpfulness-safety tradeoff compared to existing baselines.

Related papers

BSO: Safety Alignment Is Density Ratio Matching [14.960171907231663]
We show that the likelihood of the optimal safe policy admits a closed-form decomposition that reduces safety alignment to a density ratio matching problem.<n>Minimizing Bregman divergences between the data and model ratios yields Bregman Safety Optimization (BSO), a family of single-stage loss functions, that provably recover the optimal safe policy.<n> Experiments across safety alignment benchmarks show that BSO consistently improves the safety-helpfulness trade-off.
arXiv Detail & Related papers (2026-05-12T16:19:42Z)
RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs [5.100622189286672]
Fine-tuning safety-aligned language models for downstream tasks can lead to substantial degradation of refusal behavior.<n>We introduce REFUSALGUARD, a representation-level fine-tuning framework that preserves safety-relevant structure during model adaptation.
arXiv Detail & Related papers (2026-05-03T14:48:18Z)
Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines [31.031589383127677]
This paper introduces the Synthesized Guideline-based Adaptive Safety Alignment (SGASA) framework.<n>It internalizes model-generated safety guidelines to strengthen models' ability to enhance robustness against adversarial prompts.<n> experiments across multiple datasets demonstrate that SGASA significantly improves model safety, validating its adaptive and scalable effectiveness.
arXiv Detail & Related papers (2025-11-26T09:44:32Z)
SafeR-CLIP: Mitigating NSFW Content in Vision-Language Models While Preserving Pre-Trained Knowledge [51.634837361795434]
SaFeR-CLIP reconciles safety and performance, recovering up to 8.0% in zero-shot accuracy over prior methods.<n>We also contribute NSFW-Caps, a new benchmark of 1,000 highly-aligned pairs for testing safety under distributional shift.
arXiv Detail & Related papers (2025-11-20T19:00:15Z)
SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization [79.14563283347773]
Multimodal large language models (MLLMs) have demonstrated impressive reasoning and instruction-following capabilities.<n>Cross-modal couplings can produce unsafe semantics even when individual inputs are benign.<n>We propose SafeGRPO, a self-rewarded multimodal safety alignment framework.
arXiv Detail & Related papers (2025-11-17T05:09:49Z)
UpSafe$^\circ$C: Upcycling for Controllable Safety in Large Language Models [67.91151588917396]
Large Language Models (LLMs) have achieved remarkable progress across a wide range of tasks, but remain vulnerable to safety risks such as harmful content generation and jailbreak attacks.<n>We propose UpSafe$circ$C, a unified framework for enhancing LLM safety through safety-aware upcycling.<n>Our results highlight a new direction for LLM safety: moving from static alignment toward dynamic, modular, and inference-aware control.
arXiv Detail & Related papers (2025-10-02T16:43:33Z)
Shape it Up! Restoring LLM Safety during Finetuning [65.75757313781104]
Finetuning large language models (LLMs) enables user-specific customization but introduces critical safety risks.<n>We propose dynamic safety shaping (DSS), a framework that uses fine-grained safety signals to reinforce learning from safe segments of a response while suppressing unsafe content.<n>We present STAR-DSS, guided by STAR scores, that robustly mitigates finetuning risks and delivers substantial safety improvements across diverse threats, datasets, and model families.
arXiv Detail & Related papers (2025-05-22T18:05:16Z)
Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements [46.79887158348167]
The current paradigm for safety alignment of large language models (LLMs) follows a one-size-fits-all approach.<n>We propose Controllable Safety Alignment (CoSA), a framework designed to adapt models to diverse safety requirements without re-training.
arXiv Detail & Related papers (2024-10-11T16:38:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.