Learning Safety Constraints for Large Language Models

• URL: http://arxiv.org/abs/2505.24445v1
• Date: Fri, 30 May 2025 10:30:24 GMT
• Title: Learning Safety Constraints for Large Language Models
• Authors: Xin Chen, Yarden As, Andreas Krause,
• Abstract summary: Large language models (LLMs) pose significant safety risks through harmful outputs and vulnerability to adversarial attacks.<n>We propose SaP, a geometric approach to safety that learns and enforces multiple safety constraints directly in the model's representation space.<n>We develop a framework that identifies safe and unsafe regions via the polytope's facets, enabling both detection and correction of unsafe outputs.
• Score: 41.95596134688853
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have emerged as powerful tools but pose significant safety risks through harmful outputs and vulnerability to adversarial attacks. We propose SaP, short for Safety Polytope, a geometric approach to LLM safety that learns and enforces multiple safety constraints directly in the model's representation space. We develop a framework that identifies safe and unsafe regions via the polytope's facets, enabling both detection and correction of unsafe outputs through geometric steering. Unlike existing approaches that modify model weights, SaP operates post-hoc in the representation space, preserving model capabilities while enforcing safety constraints. Experiments across multiple LLMs demonstrate that our method can effectively detect unethical inputs, reduce adversarial attack success rates while maintaining performance on standard tasks, thus highlighting the importance of having an explicit geometric model for safety. Analysis of the learned polytope facets reveals emergence of specialization in detecting different semantic notions of safety, providing interpretable insights into how safety is captured in LLMs' representation space.

Related papers

• ReGA: Representation-Guided Abstraction for Model-based Safeguarding of LLMs [0.9285458070502282]
  Large Language Models (LLMs) have achieved significant success in various tasks, yet concerns about their safety and security have emerged.<n>To analyze and monitor machine learning models, model-based analysis has demonstrated notable potential in stateful deep neural networks.<n>We propose ReGA, a model-based analysis framework with representation-guided abstraction, to safeguard LLMs against harmful prompts and generations.
  arXiv  Detail & Related papers  (2025-06-02T15:17:38Z)
• Shape it Up! Restoring LLM Safety during Finetuning [66.46166656543761]
  Finetuning large language models (LLMs) enables user-specific customization but introduces critical safety risks.<n>We propose dynamic safety shaping (DSS), a framework that uses fine-grained safety signals to reinforce learning from safe segments of a response while suppressing unsafe content.<n>We present STAR-DSS, guided by STAR scores, that robustly mitigates finetuning risks and delivers substantial safety improvements across diverse threats, datasets, and model families.
  arXiv  Detail & Related papers  (2025-05-22T18:05:16Z)
• Safety Alignment Can Be Not Superficial With Explicit Safety Signals [8.297367440457508]
  Recent studies on the safety alignment of large language models (LLMs) have revealed that existing approaches often operate superficially.<n>This paper identifies a fundamental cause of this superficiality: existing alignment approaches presume that models can implicitly learn a safety-related reasoning task during the alignment process.<n>By explicitly introducing a safety-related binary classification task and integrating its signals with our attention and decoding strategies, we eliminate this ambiguity.
  arXiv  Detail & Related papers  (2025-05-19T20:40:46Z)
• On Almost Surely Safe Alignment of Large Language Models at Inference-Time [20.5164976103514]
  We introduce a novel inference-time alignment approach for LLMs that aims to generate safe responses almost surely.<n>We augment a safety state that tracks the evolution of safety constraints and dynamically penalizes unsafe generations.<n>We demonstrate formal safety guarantees w.r.t. the given cost model upon solving the MDP in the latent space with sufficiently large penalties.
  arXiv  Detail & Related papers  (2025-02-03T09:59:32Z)
• Towards Inference-time Category-wise Safety Steering for Large Language Models [3.712541089289745]
  Large language models (LLMs) have seen unprecedented advancements in capabilities and applications across a variety of use-cases. The fragile nature of LLMs warrants additional safety steering steps via training-free, inference-time methods. Unlike recent inference-time safety steering works, in this paper we explore safety steering of LLM outputs using category-specific steering vectors.
  arXiv  Detail & Related papers  (2024-10-02T02:02:06Z)
• SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering [56.92068213969036]
  Safety alignment is indispensable for Large Language Models (LLMs) to defend threats from malicious instructions.<n>Recent researches reveal safety-aligned LLMs prone to reject benign queries due to the exaggerated safety issue.<n>We propose a Safety-Conscious Activation Steering (SCANS) method to mitigate the exaggerated safety concerns.
  arXiv  Detail & Related papers  (2024-08-21T10:01:34Z)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models [65.06446825020578]
  Safety alignment is crucial to ensure that large language models (LLMs) behave in ways that align with human preferences and prevent harmful actions during inference. We aim to measure the risks in finetuning LLMs through navigating the LLM safety landscape.
  arXiv  Detail & Related papers  (2024-05-27T17:31:56Z)
• The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness [56.174255970895466]
  Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications. This paper presents Safety and Over-Defensiveness Evaluation (SODE) benchmark.
  arXiv  Detail & Related papers  (2023-12-30T17:37:06Z)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis [63.532413807686524]
  This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL) We propose a new architecture that handles the trade-off between efficient progress and safety during exploration.
  arXiv  Detail & Related papers  (2023-12-18T16:09:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.