Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions

• URL: http://arxiv.org/abs/2509.23437v1
• Date: Sat, 27 Sep 2025 18:12:35 GMT
• Title: Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions
• Authors: Steve Hong, Runa Eschenhagen, Bruno Mlodozeniec, Richard Turner,
• Abstract summary: We investigate the effect of Hessian approximation quality on influence-function attributions in a controlled classification setting.<n>Our experiments show that better Hessian approximations consistently yield better influence score quality.<n>We further decompose the approximation steps for recent Hessian approximation methods and evaluate each step's influence on attribution accuracy.
• Score: 5.937280131734114
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Influence functions offer a principled way to trace model predictions back to training data, but their use in deep learning is hampered by the need to invert a large, ill-conditioned Hessian matrix. Approximations such as Generalised Gauss-Newton (GGN) and Kronecker-Factored Approximate Curvature (K-FAC) have been proposed to make influence computation tractable, yet it remains unclear how the departure from exactness impacts data attribution performance. Critically, given the restricted regime in which influence functions are derived, it is not necessarily clear better Hessian approximations should even lead to better data attribution performance. In this paper, we investigate the effect of Hessian approximation quality on influence-function attributions in a controlled classification setting. Our experiments show that better Hessian approximations consistently yield better influence score quality, offering justification for recent research efforts towards that end. We further decompose the approximation steps for recent Hessian approximation methods and evaluate each step's influence on attribution accuracy. Notably, the mismatch between K-FAC eigenvalues and GGN/EK-FAC eigenvalues accounts for the majority of the error and influence loss. These findings highlight which approximations are most critical, guiding future efforts to balance computational tractability and attribution accuracy.

Related papers

• Z0-Inf: Zeroth Order Approximation for Data Influence [47.682602051124235]
  We introduce a highly efficient zeroth-order approximation for estimating the influence of training data.<n>Our approach achieves superior accuracy in estimating self-influence and comparable or improved accuracy in estimating train-test influence for fine-tuned large language models.
  arXiv  Detail & Related papers  (2025-10-13T18:30:37Z)
• Understanding Data Influence with Differential Approximation [63.817689230826595]
  We introduce a new formulation to approximate a sample's influence by accumulating the differences in influence between consecutive learning steps, which we term Diff-In.<n>By employing second-order approximations, we approximate these difference terms with high accuracy while eliminating the need for model convexity required by existing methods.<n>Our theoretical analysis demonstrates that Diff-In achieves significantly lower approximation error compared to existing influence estimators.
  arXiv  Detail & Related papers  (2025-08-20T11:59:32Z)
• Revisiting Data Attribution for Influence Functions [13.88866465448849]
  This paper comprehensively reviews the data attribution capability of influence functions in deep learning.<n>We discuss their theoretical foundations, recent algorithmic advances for efficient inverse-Hessian-vector product estimation, and evaluate their effectiveness for data attribution and mislabel detection.
  arXiv  Detail & Related papers  (2025-08-10T11:15:07Z)
• Detecting Instruction Fine-tuning Attack on Language Models with Influence Function [6.760293300577228]
  Instruction fine-tuning attacks undermine model alignment and pose security risks in real-world deployment.<n>We present a simple and effective approach to detect and mitigate such attacks using influence functions.<n>We are the first to apply influence functions for detecting language model instruction fine-tuning attacks on large-scale datasets.
  arXiv  Detail & Related papers  (2025-04-12T00:50:28Z)
• C-XGBoost: A tree boosting model for causal effect estimation [8.246161706153805]
  Causal effect estimation aims at estimating the Average Treatment Effect as well as the Conditional Average Treatment Effect of a treatment to an outcome from the available data. We propose a new causal inference model, named C-XGBoost, for the prediction of potential outcomes.
  arXiv  Detail & Related papers  (2024-03-31T17:43:37Z)
• B-Learner: Quasi-Oracle Bounds on Heterogeneous Causal Effects Under Hidden Confounding [51.74479522965712]
  We propose a meta-learner called the B-Learner, which can efficiently learn sharp bounds on the CATE function under limits on hidden confounding. We prove its estimates are valid, sharp, efficient, and have a quasi-oracle property with respect to the constituent estimators under more general conditions than existing methods.
  arXiv  Detail & Related papers  (2023-04-20T18:07:19Z)
• Data-Driven Influence Functions for Optimization-Based Causal Inference [105.5385525290466]
  We study a constructive algorithm that approximates Gateaux derivatives for statistical functionals by finite differencing. We study the case where probability distributions are not known a priori but need to be estimated from data.
  arXiv  Detail & Related papers  (2022-08-29T16:16:22Z)
• Inf-CP: A Reliable Channel Pruning based on Channel Influence [4.692400531340393]
  One of the most effective methods of channel pruning is to trim on the basis of the importance of each neuron. Previous works have proposed to trim by considering the statistics of a single layer or a plurality of successive layers of neurons. We propose to use ensemble learning to train a model for different batches of data.
  arXiv  Detail & Related papers  (2021-12-05T09:30:43Z)
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation [85.75516599931632]
  We study the off-policy evaluation problem in reinforcement learning with linear function approximation. We propose an algorithm, VA-OPE, which uses the estimated variance of the value function to reweight the Bellman residual in Fitted Q-Iteration.
  arXiv  Detail & Related papers  (2021-06-22T17:58:46Z)
• Influence Functions in Deep Learning Are Fragile [52.31375893260445]
  influence functions approximate the effect of samples in test-time predictions. influence estimates are fairly accurate for shallow networks. Hessian regularization is important to get highquality influence estimates.
  arXiv  Detail & Related papers  (2020-06-25T18:25:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.