Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations

• URL: http://arxiv.org/abs/2406.11801v1
• Date: Mon, 17 Jun 2024 17:48:13 GMT
• Title: Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations
• Authors: Rima Hazra, Sayan Layek, Somnath Banerjee, Soujanya Poria,
• Abstract summary: Current alignment methods struggle with dynamic user intentions and complex objectives. We propose Safety Arithmetic, a training-free framework enhancing safety across different scenarios. Our experiments show that Safety Arithmetic significantly improves safety measures, reduces over-safety, and maintains model utility.
• Score: 19.132597762214722
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ensuring the safe alignment of large language models (LLMs) with human values is critical as they become integral to applications like translation and question answering. Current alignment methods struggle with dynamic user intentions and complex objectives, making models vulnerable to generating harmful content. We propose Safety Arithmetic, a training-free framework enhancing LLM safety across different scenarios: Base models, Supervised fine-tuned models (SFT), and Edited models. Safety Arithmetic involves Harm Direction Removal to avoid harmful content and Safety Alignment to promote safe responses. Additionally, we present NoIntentEdit, a dataset highlighting edit instances that could compromise model safety if used unintentionally. Our experiments show that Safety Arithmetic significantly improves safety measures, reduces over-safety, and maintains model utility, outperforming existing methods in ensuring safe content generation.

Related papers

• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study [64.9691741899956]
  Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. We design a synthetic data generation framework that captures salient aspects of an unsafe input. Using this, we investigate three well-known safety fine-tuning methods.
  arXiv  Detail & Related papers  (2024-07-14T16:12:57Z)
• Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training [67.30423823744506]
  This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful
  arXiv  Detail & Related papers  (2024-07-12T09:36:33Z)
• SafeInfer: Context Adaptive Decoding Time Safety Alignment for Large Language Models [5.6874111521946356]
  Safety-aligned language models often exhibit fragile and imbalanced safety mechanisms. We propose SafeInfer, a context-adaptive, decoding-time safety alignment strategy. HarmEval is a novel benchmark for extensive safety evaluations.
  arXiv  Detail & Related papers  (2024-06-18T05:03:23Z)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models [65.06446825020578]
  We aim to measure the risks in finetuning large language models (LLMs) through navigating the safety landscape. We discover a new phenomenon observed universally in the model parameter space of popular open-source LLMs. Visualizing the safety landscape of the aligned model enables us to understand how finetuning compromises safety by dragging the model away from the safety basin.
  arXiv  Detail & Related papers  (2024-05-27T17:31:56Z)
• Towards Comprehensive and Efficient Post Safety Alignment of Large Language Models via Safety Patching [77.36097118561057]
  textscSafePatching is a novel framework for comprehensive and efficient PSA. textscSafePatching achieves a more comprehensive and efficient PSA than baseline methods.
  arXiv  Detail & Related papers  (2024-05-22T16:51:07Z)
• Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints [15.904640266226023]
  We design a safety model that performs credit assignment to assess contributions of partial state-action trajectories on safety. We derive an effective algorithm for optimizing a safe policy using the learned safety model. We devise a method to dynamically adapt the tradeoff coefficient between safety reward and safety compliance.
  arXiv  Detail & Related papers  (2024-05-05T17:27:22Z)
• Towards Safety and Helpfulness Balanced Responses via Controllable Large Language Models [64.5204594279587]
  A model that prioritizes safety will cause users to feel less engaged and assisted while prioritizing helpfulness will potentially cause harm. We propose to balance safety and helpfulness in diverse use cases by controlling both attributes in large language models.
  arXiv  Detail & Related papers  (2024-04-01T17:59:06Z)
• The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness [56.174255970895466]
  Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications. This paper presents Safety and Over-Defensiveness Evaluation (SODE) benchmark.
  arXiv  Detail & Related papers  (2023-12-30T17:37:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.