Towards Inference-time Category-wise Safety Steering for Large Language Models

• URL: http://arxiv.org/abs/2410.01174v1
• Date: Wed, 2 Oct 2024 02:02:06 GMT
• Title: Towards Inference-time Category-wise Safety Steering for Large Language Models
• Authors: Amrita Bhattacharjee, Shaona Ghosh, Traian Rebedea, Christopher Parisien,
• Abstract summary: Large language models (LLMs) have seen unprecedented advancements in capabilities and applications across a variety of use-cases. The fragile nature of LLMs warrants additional safety steering steps via training-free, inference-time methods. Unlike recent inference-time safety steering works, in this paper we explore safety steering of LLM outputs using category-specific steering vectors.
• Score: 3.712541089289745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While large language models (LLMs) have seen unprecedented advancements in capabilities and applications across a variety of use-cases, safety alignment of these models is still an area of active research. The fragile nature of LLMs, even models that have undergone extensive alignment and safety training regimes, warrants additional safety steering steps via training-free, inference-time methods. While recent work in the area of mechanistic interpretability has investigated how activations in latent representation spaces may encode concepts, and thereafter performed representation engineering to induce such concepts in LLM outputs, the applicability of such for safety is relatively under-explored. Unlike recent inference-time safety steering works, in this paper we explore safety steering of LLM outputs using: (i) category-specific steering vectors, thereby enabling fine-grained control over the steering, and (ii) sophisticated methods for extracting informative steering vectors for more effective safety steering while retaining quality of the generated text. We demonstrate our exploration on multiple LLMs and datasets, and showcase the effectiveness of the proposed steering method, along with a discussion on the implications and best practices.

Related papers

• Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarios [32.16984263644299]
  Large Language Models (LLMs) can generate valuable data for safety measures, but often exhibit distributional biases. We propose a novel framework that integrates active learning with clustering to guide LLM generation. Our results show that the proposed framework produces a more representative set of safety scenarios without requiring prior knowledge of the underlying data distribution.
  arXiv  Detail & Related papers  (2024-10-14T21:48:14Z)
• Nothing in Excess: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering [56.92068213969036]
  Safety alignment is indispensable for Large language models (LLMs) to defend threats from malicious instructions. Recent researches reveal safety-aligned LLMs prone to reject benign queries due to the exaggerated safety issue. We propose a Safety-Conscious Activation Steering (SCANS) method to mitigate the exaggerated safety concerns.
  arXiv  Detail & Related papers  (2024-08-21T10:01:34Z)
• Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations [76.19419888353586]
  Large language models (LLMs) are susceptible to a variety of risks, from non-faithful output to biased and toxic generations. We present our efforts to create and deploy a library of detectors: compact and easy-to-build classification models that provide labels for various harms.
  arXiv  Detail & Related papers  (2024-03-09T21:07:16Z)
• The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness [56.174255970895466]
  Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications. This paper presents Safety and Over-Defensiveness Evaluation (SODE) benchmark.
  arXiv  Detail & Related papers  (2023-12-30T17:37:06Z)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis [63.532413807686524]
  This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL) We propose a new architecture that handles the trade-off between efficient progress and safety during exploration.
  arXiv  Detail & Related papers  (2023-12-18T16:09:43Z)
• Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
  This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems. LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
  arXiv  Detail & Related papers  (2023-11-28T03:13:09Z)
• Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks [70.76757529955577]
  This paper revisits prior work in this scope from the perspective of state-wise safe RL. We propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit.
  arXiv  Detail & Related papers  (2022-12-12T06:30:17Z)
• Safe Model-Based Reinforcement Learning Using Robust Control Barrier Functions [43.713259595810854]
  An increasingly common approach to address safety involves the addition of a safety layer that projects the RL actions onto a safe set of actions. In this paper, we frame safety as a differentiable robust-control-barrier-function layer in a model-based RL framework.
  arXiv  Detail & Related papers  (2021-10-11T17:00:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.