Detecting Instruction Fine-tuning Attack on Language Models with Influence Function

• URL: http://arxiv.org/abs/2504.09026v1
• Date: Sat, 12 Apr 2025 00:50:28 GMT
• Title: Detecting Instruction Fine-tuning Attack on Language Models with Influence Function
• Authors: Jiawei Li,
• Abstract summary: Instruction fine-tuning attacks undermine model alignment and pose security risks in real-world deployment.<n>We present a simple and effective approach to detect and mitigate such attacks using influence functions.<n>We are the first to apply influence functions for detecting language model instruction fine-tuning attacks on large-scale datasets.
• Score: 6.760293300577228
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Instruction fine-tuning attacks pose a significant threat to large language models (LLMs) by subtly embedding poisoned data in fine-tuning datasets, which can trigger harmful or unintended responses across a range of tasks. This undermines model alignment and poses security risks in real-world deployment. In this work, we present a simple and effective approach to detect and mitigate such attacks using influence functions, a classical statistical tool adapted for machine learning interpretation. Traditionally, the high computational costs of influence functions have limited their application to large models and datasets. The recent Eigenvalue-Corrected Kronecker-Factored Approximate Curvature (EK-FAC) approximation method enables efficient influence score computation, making it feasible for large-scale analysis. We are the first to apply influence functions for detecting language model instruction fine-tuning attacks on large-scale datasets, as both the instruction fine-tuning attack on language models and the influence calculation approximation technique are relatively new. Our large-scale empirical evaluation of influence functions on 50,000 fine-tuning examples and 32 tasks reveals a strong association between influence scores and sentiment. Building on this, we introduce a novel sentiment transformation combined with influence functions to detect and remove critical poisons -- poisoned data points that skew model predictions. Removing these poisons (only 1% of total data) recovers model performance to near-clean levels, demonstrating the effectiveness and efficiency of our approach. Artifact is available at https://github.com/lijiawei20161002/Poison-Detection. WARNING: This paper contains offensive data examples.

Related papers

• Towards Model Resistant to Transferable Adversarial Examples via Trigger Activation [95.3977252782181]
  Adversarial examples, characterized by imperceptible perturbations, pose significant threats to deep neural networks by misleading their predictions. We introduce a novel training paradigm aimed at enhancing robustness against transferable adversarial examples (TAEs) in a more efficient and effective way.
  arXiv  Detail & Related papers  (2025-04-20T09:07:10Z)
• Revisit, Extend, and Enhance Hessian-Free Influence Functions [26.105554752277648]
  Influence functions serve as crucial tools for assessing sample influence in model interpretation, subset training set selection, and more. In this paper, we revisit a specific, albeit effective approximation method known as Trac. This method substitutes the inverse of the Hessian matrix with an identity matrix.
  arXiv  Detail & Related papers  (2024-05-25T03:43:36Z)
• Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models [36.05242956018461]
  In this paper, we establish a bridge between identifying detrimental training samples via influence functions and outlier gradient detection. We first validate the hypothesis of our proposed outlier gradient analysis approach on synthetic datasets. We then demonstrate its effectiveness in detecting mislabeled samples in vision models and selecting data samples for improving performance of natural language processing transformer models.
  arXiv  Detail & Related papers  (2024-05-06T21:34:46Z)
• C-XGBoost: A tree boosting model for causal effect estimation [8.246161706153805]
  Causal effect estimation aims at estimating the Average Treatment Effect as well as the Conditional Average Treatment Effect of a treatment to an outcome from the available data. We propose a new causal inference model, named C-XGBoost, for the prediction of potential outcomes.
  arXiv  Detail & Related papers  (2024-03-31T17:43:37Z)
• DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models [31.65198592956842]
  We propose DataInf, an efficient influence approximation method that is practical for large-scale generative AI models. Our theoretical analysis shows that DataInf is particularly well-suited for parameter-efficient fine-tuning techniques such as LoRA. In applications to RoBERTa-large, Llama-2-13B-chat, and stable-diffusion-v1.5 models, DataInf effectively identifies the most influential fine-tuning examples better than other approximate influence scores.
  arXiv  Detail & Related papers  (2023-10-02T04:59:19Z)
• Studying Large Language Model Generalization with Influence Functions [29.577692176892135]
  Influence functions aim to answer a counterfactual: how would the model's parameters (and hence its outputs) change if a sequence were added to the training set? We use the Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) approximation to scale influence functions up to large language models (LLMs) with up to 52 billion parameters. We investigate generalization patterns of LLMs, including the sparsity of the influence patterns, increasing abstraction with scale, math and programming abilities, cross-lingual generalization, and role-playing behavior.
  arXiv  Detail & Related papers  (2023-08-07T04:47:42Z)
• Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions [59.284907093349425]
  Large amounts of training data are one of the major reasons for the high performance of state-of-the-art NLP models. We provide a language for describing how training data influences predictions, through a causal framework. Our framework bypasses the need to retrain expensive models and allows us to estimate causal effects based on observational data alone.
  arXiv  Detail & Related papers  (2022-07-28T17:36:24Z)
• CARLA-GeAR: a Dataset Generator for a Systematic Evaluation of Adversarial Robustness of Vision Models [61.68061613161187]
  This paper presents CARLA-GeAR, a tool for the automatic generation of synthetic datasets for evaluating the robustness of neural models against physical adversarial patches. The tool is built on the CARLA simulator, using its Python API, and allows the generation of datasets for several vision tasks in the context of autonomous driving. The paper presents an experimental study to evaluate the performance of some defense methods against such attacks, showing how the datasets generated with CARLA-GeAR might be used in future work as a benchmark for adversarial defense in the real world.
  arXiv  Detail & Related papers  (2022-06-09T09:17:38Z)
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging [112.19994766375231]
  Influence functions approximate the 'influences' of training data-points for test predictions. We present FastIF, a set of simple modifications to influence functions that significantly improves their run-time. Our experiments demonstrate the potential of influence functions in model interpretation and correcting model errors.
  arXiv  Detail & Related papers  (2020-12-31T18:02:34Z)
• Influence Functions in Deep Learning Are Fragile [52.31375893260445]
  influence functions approximate the effect of samples in test-time predictions. influence estimates are fairly accurate for shallow networks. Hessian regularization is important to get highquality influence estimates.
  arXiv  Detail & Related papers  (2020-06-25T18:25:59Z)
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions [55.660255727031725]
  Influence functions explain the decisions of a model by identifying influential training examples. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. We develop a new measure based on influence functions that can reveal artifacts in training data.
  arXiv  Detail & Related papers  (2020-05-14T00:45:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.