Safety-Aware Fine-Tuning of Large Language Models

• URL: http://arxiv.org/abs/2410.10014v1
• Date: Sun, 13 Oct 2024 21:24:25 GMT
• Title: Safety-Aware Fine-Tuning of Large Language Models
• Authors: Hyeong Kyu Choi, Xuefeng Du, Yixuan Li,
• Abstract summary: Fine-tuning Large Language Models (LLMs) has emerged as a common practice for tailoring models to individual needs and preferences. We propose a novel Safety-Aware Fine-Tuning (SAFT) framework designed to automatically detect and remove potentially harmful data.
• Score: 29.5636201427693
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning Large Language Models (LLMs) has emerged as a common practice for tailoring models to individual needs and preferences. The choice of datasets for fine-tuning can be diverse, introducing safety concerns regarding the potential inclusion of harmful data samples. Manually filtering or avoiding such samples, however, can be labor-intensive and subjective. To address these difficulties, we propose a novel Safety-Aware Fine-Tuning (SAFT) framework designed to automatically detect and remove potentially harmful data, by leveraging a scoring function that exploits the subspace information of harmful and benign samples. Experimental results demonstrate the efficacy of SAFT across different LLMs and varying contamination rates, achieving reductions in harmfulness of up to 27.8%. Going beyond, we delve into the mechanism of our approach and validate its versatility in addressing practical challenges in real-world scenarios.

Related papers

• Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models [2.3281513013731145]
  Fine-tuning large language models (LLMs) for specific tasks introduces privacy risks, as models may inadvertently memorise and leak sensitive training data. Differential Privacy (DP) offers a solution to mitigate these risks, but introduces significant computational and performance trade-offs. We show that PEFT methods achieve comparable performance to standard fine-tuning while requiring fewer parameters and significantly reducing privacy leakage.
  arXiv  Detail & Related papers  (2024-11-24T13:17:36Z)
• Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset [4.522849055040843]
  This study audited the Helpful and Harmless dataset by Anthropic. Our findings highlight the need for more nuanced, context-sensitive approaches to safety mitigation in large language models.
  arXiv  Detail & Related papers  (2024-11-12T23:43:20Z)
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study [64.9691741899956]
  Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. We design a synthetic data generation framework that captures salient aspects of an unsafe input. Using this, we investigate three well-known safety fine-tuning methods.
  arXiv  Detail & Related papers  (2024-07-14T16:12:57Z)
• Quantifying Prediction Consistency Under Model Multiplicity in Tabular LLMs [10.494477811252034]
  Fine-tuning large language models can lead to textitfine-tuning multiplicity, where equally well-performing models make conflicting predictions on the same inputs. This raises critical concerns about the robustness and reliability of Tabular LLMs. This work proposes a novel metric to quantify the robustness of individual predictions without expensive model retraining.
  arXiv  Detail & Related papers  (2024-07-04T22:22:09Z)
• Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models [89.88010750772413]
  Synthetic data has been proposed as a solution to address the issue of high-quality data scarcity in the training of large language models (LLMs) Our work delves into these specific flaws associated with question-answer (Q-A) pairs, a prevalent type of synthetic data, and presents a method based on unlearning techniques to mitigate these flaws. Our work has yielded key insights into the effective use of synthetic data, aiming to promote more robust and efficient LLM training.
  arXiv  Detail & Related papers  (2024-06-18T08:38:59Z)
• Mimicking User Data: On Mitigating Fine-Tuning Risks in Closed Large Language Models [53.50543146583101]
  Fine-tuning large language models on small datasets can enhance their performance on specific downstream tasks. Malicious actors can subtly manipulate the structure of almost any task-specific dataset to foster significantly more dangerous model behaviors. We propose a novel mitigation strategy that mixes in safety data which mimics the task format and prompting style of the user data.
  arXiv  Detail & Related papers  (2024-06-12T18:33:11Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness [50.52507648690234]
  Federated learning has the risk of skewing fine-tuning features and compromising the robustness of the model. We introduce three robustness indicators and conduct experiments across diverse robust datasets. Our approach markedly enhances the robustness across diverse scenarios, encompassing various parameter-efficient fine-tuning methods.
  arXiv  Detail & Related papers  (2024-01-25T09:18:51Z)
• Unveiling Safety Vulnerabilities of Large Language Models [4.562678399685183]
  This paper introduces a unique dataset containing adversarial examples in the form of questions, which we call AttaQ. We assess the efficacy of our dataset by analyzing the vulnerabilities of various models when subjected to it. We introduce a novel automatic approach for identifying and naming vulnerable semantic regions.
  arXiv  Detail & Related papers  (2023-11-07T16:50:33Z)
• Flexible and Robust Counterfactual Explanations with Minimal Satisfiable Perturbations [56.941276017696076]
  We propose a conceptually simple yet effective solution named Counterfactual Explanations with Minimal Satisfiable Perturbations (CEMSP) CEMSP constrains changing values of abnormal features with the help of their semantically meaningful normal ranges. Compared to existing methods, we conduct comprehensive experiments on both synthetic and real-world datasets to demonstrate that our method provides more robust explanations while preserving flexibility.
  arXiv  Detail & Related papers  (2023-09-09T04:05:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.