Towards Context-Invariant Safety Alignment for Large Language Models

• URL: http://arxiv.org/abs/2605.20994v1
• Date: Wed, 20 May 2026 10:33:11 GMT
• Title: Towards Context-Invariant Safety Alignment for Large Language Models
• Abstract summary: We introduce Anchor Invariance Regularization (AIR), which treats verifiable prompts as anchors and uses a stop-gradient target to regularize only the open-ended variants toward the anchor performance.<n>AIR boosts in-distribution group accuracy by 12.71% and out-of-distribution consistency by 33.49%, making safety constraints robust to adversarial framings.
• Score: 37.23800025875439
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Preference-based post-training aligns LLMs with human intent, yet safety behavior often remains brittle. A model may refuse a harmful request in a standard prompt but comply when the same intent is wrapped in adversarial wording. We suggest that robust safety requires context-invariant alignment, where behavior depends on the underlying intent rather than surface form. Enforcing invariance is difficult in alignment because not all training signals are equally trustworthy; for some prompt variants we can obtain verifiable feedback (e.g., multiple-choice), while for open-ended variants we typically rely on noisy, gameable reward proxies (e.g., learned judges). As a result, standard symmetric invariance regularizers can reduce cross-context discrepancies by lowering performance on reliable variants instead of improving open-ended robustness. To address this, we introduce Anchor Invariance Regularization (AIR), which treats verifiable prompts as anchors and uses a stop-gradient target to regularize only the open-ended variants toward the anchor performance. AIR is implemented as a plug-in auxiliary loss and combined with group-based preference optimization (e.g., GRPO) via heterogeneous prompt grouping. Across Safety, Moral Reasoning, and Math, AIR improves context invariance, boosting in-distribution group accuracy by 12.71% and out-of-distribution consistency by 33.49%, making safety constraints robust to adversarial framings.

Related papers

• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors [48.881343993730844]
  Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines.<n> adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present.<n>Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors.
  arXiv  Detail & Related papers  (2026-04-14T06:48:33Z)
• Beyond Corner Patches: Semantics-Aware Backdoor Attack in Federated Learning [6.76324539337304]
  Backdoor attacks on federated learning (FL) are most often evaluated with synthetic corner patches or out-of-distribution patterns.<n>We propose SABLE, a Semantics-Aware Backdoor for LEarning in federated settings.<n>Our semantics-driven triggers achieve high targeted attack success rates while preserving benign test accuracy.
  arXiv  Detail & Related papers  (2026-03-31T06:48:25Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding [59.16244104797919]
  We propose **Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC)**.<n>DARC frames response selection as distributionally robust, risk-sensitive decision making.<n>Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality.
  arXiv  Detail & Related papers  (2026-03-09T09:21:29Z)
• Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing [37.86964394689924]
  We introduce the Consensus Clustering LinUCB Bandit (CCLUB), a unified framework for adaptive social alignment via system-prompt routing.<n>CCLUB employs a conservative consensus clustering mechanism, effectively preventing unsafe generalization across semantically proximal but risk-divergent contexts.<n>Our theoretical analysis yields a sublinear regret guarantee, demonstrating near-optimal performance of CCLUB.
  arXiv  Detail & Related papers  (2026-03-04T08:00:16Z)
• Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment [6.428964221372943]
  We propose Geometric Anchor Preference Optimization (GAPO), which replaces the fixed reference with a dynamic, geometry-aware anchor.<n>GAPO consistently improves robustness while matching or improving performance on standard LLM alignment and reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-02-04T00:40:21Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.