PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models

• URL: http://arxiv.org/abs/2606.25442v1
• Date: Wed, 24 Jun 2026 06:10:33 GMT
• Title: PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models
• Abstract summary: In real-world deployment, emerging safety requirements are often specified as natural-language policies.<n>This creates a mismatch between rapidly evolving safety policies and conventional data-driven alignment methods.<n>We propose PolicyAlign, a framework for directly aligning large language models with safety policies.
• Score: 19.807112285121715
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Safety alignment of large language models (LLMs) typically depends on high-quality supervision data, such as safe demonstrations or preference pairs. However, in real-world deployment, emerging safety requirements are often specified as natural-language policies, while corresponding supervision data may be costly, delayed, or unavailable. This creates a mismatch between rapidly evolving safety policies and conventional data-driven alignment methods. To address this, we propose PolicyAlign, a simple yet effective framework for directly aligning LLMs with safety policies. Given a safety policy, PolicyAlign first synthesizes policy-violating instructions and then performs on-policy self-distillation to internalize policy-guided behavior. To improve training stability and data efficiency, we further introduce Policy-Sensitive Filtering, which selects instructions where the policy induces the largest behavioral shift. Experiments across multiple models show that PolicyAlign consistently improves safety while maintaining low over-refusal and preserving general capabilities. PolicyAlign also generalizes to medical, legal, and financial safety scenarios, highlighting its potential as a scalable and maintainable approach to policy-based LLM safety alignment. The code is released at https://github.com/Qwen-Applications/PolicyAlign.

Related papers

• PolicyMem: Geometric Policy Memory for LLM Governance [72.04962480384322]
  We introduce PolicyMem, a geometric policy memory that externalizes natural-language policies as reusable geometric memory objects.<n>A memory writer compiles natural-language policies into policy memory slots, and query-response pairs read the policy memory through projection energy.<n>The resulting policy-evidence profile directly mediates the safety verdict and is reused for policy attribution and post-intervention verification.
  arXiv  Detail & Related papers  (2026-09-12T06:07:43Z)
• Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications [34.429470292997145]
  We introduce a novel framework that brings the rigor of specification-based software testing to AI safety.<n>POLARIS first compiles unstructured natural-language policies into First-Order Logic representations, establishing a traceable link between high-level rules and concrete test cases.<n>This formalization enables the construction of a Semantic Policy Graph, where complex policy violation scenarios are encoded as traversable paths.<n>Experiments demonstrate that POLARIS achieves higher policy coverage and attack success counts compared to established baselines.
  arXiv  Detail & Related papers  (2026-05-24T05:51:23Z)
• SafeAdapt: Provably Safe Policy Updates in Deep Reinforcement Learning [15.190197265099748]
  Safety guarantees are a prerequisite to the deployment of reinforcement learning (RL) agents in safety-critical tasks.<n>Current approaches either do not provide formal guarantees or verify policy safety only a posteriori.<n>We propose a novel a priori approach to safe policy updates in continual RL by introducing the Rashomon set.<n>We show that one can provide formal, provable guarantees for arbitrary RL algorithms used to update a policy by projecting their updates onto the Rashomon set.
  arXiv  Detail & Related papers  (2026-04-10T16:09:39Z)
• Towards Policy-Adaptive Image Guardrail: Benchmark and Method [21.041111216560545]
  Vision-language models (VLMs) offer a more adaptable and generalizable foundation for dynamic safety guardrails.<n>Existing VLM-based safeguarding methods are typically trained and evaluated under only a fixed safety policy.<n>We introduce SafeGuard-VL, a reinforcement learning-based method with verifiable rewards for robust unsafe-image guardrails.
  arXiv  Detail & Related papers  (2026-03-01T18:59:21Z)
• Beyond Static Alignment: Hierarchical Policy Control for LLM Safety via Risk-Aware Chain-of-Thought [5.251527748612469]
  Large Language Models (LLMs) face a fundamental safety-helpfulness trade-off due to static, one-size-fits-all safety policies.<n>We present textbfPACT (Prompt-Thought Action via Chain-of-Thought), a framework for dynamic safety control through explicit, risk-aware reasoning.
  arXiv  Detail & Related papers  (2026-02-06T12:20:01Z)
• EXPO: Stable Reinforcement Learning with Expressive Policies [74.30151915786233]
  We propose a sample-efficient online reinforcement learning algorithm to maximize value with two parameterized policies.<n>Our approach yields up to 2-3x improvement in sample efficiency on average over prior methods.
  arXiv  Detail & Related papers  (2025-07-10T17:57:46Z)
• CSPI-MT: Calibrated Safe Policy Improvement with Multiple Testing for Threshold Policies [30.57323631122579]
  We focus on threshold policies, a ubiquitous class of policies with applications in economics, healthcare, and digital advertising. Existing methods rely on potentially underpowered safety checks and limit the opportunities for finding safe improvements. We show that in adversarial settings, our approach controls the rate of adopting a policy worse than the baseline to the pre-specified error level.
  arXiv  Detail & Related papers  (2024-08-21T21:38:03Z)
• Offline Reinforcement Learning with Closed-Form Policy Improvement Operators [88.54210578912554]
  Behavior constrained policy optimization has been demonstrated to be a successful paradigm for tackling Offline Reinforcement Learning. In this paper, we propose our closed-form policy improvement operators. We empirically demonstrate their effectiveness over state-of-the-art algorithms on the standard D4RL benchmark.
  arXiv  Detail & Related papers  (2022-11-29T06:29:26Z)
• Mutual Information Regularized Offline Reinforcement Learning [76.05299071490913]
  We propose a novel MISA framework to approach offline RL from the perspective of Mutual Information between States and Actions in the dataset. We show that optimizing this lower bound is equivalent to maximizing the likelihood of a one-step improved policy on the offline dataset. We introduce 3 different variants of MISA, and empirically demonstrate that tighter mutual information lower bound gives better offline RL performance.
  arXiv  Detail & Related papers  (2022-10-14T03:22:43Z)
• Safe Reinforcement Learning via Confidence-Based Filters [78.39359694273575]
  We develop a control-theoretic approach for certifying state safety constraints for nominal policies learned via standard reinforcement learning techniques. We provide formal safety guarantees, and empirically demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2022-07-04T11:43:23Z)
• Memory-Constrained Policy Optimization [59.63021433336966]
  We introduce a new constrained optimization method for policy gradient reinforcement learning. We form a second trust region through the construction of another virtual policy that represents a wide range of past policies. We then enforce the new policy to stay closer to the virtual policy, which is beneficial in case the old policy performs badly.
  arXiv  Detail & Related papers  (2022-04-20T08:50:23Z)
• Preventing Imitation Learning with Adversarial Policy Ensembles [79.81807680370677]
  Imitation learning can reproduce policies by observing experts, which poses a problem regarding policy privacy. How can we protect against external observers cloning our proprietary policies? We introduce a new reinforcement learning framework, where we train an ensemble of near-optimal policies.
  arXiv  Detail & Related papers  (2020-01-31T01:57:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.