Adapting and Evaluating Influence-Estimation Methods for Gradient-Boosted Decision Trees

• URL: http://arxiv.org/abs/2205.00359v3
• Date: Wed, 31 May 2023 04:25:04 GMT
• Title: Adapting and Evaluating Influence-Estimation Methods for Gradient-Boosted Decision Trees
• Authors: Jonathan Brophy, Zayd Hammoudeh, and Daniel Lowd
• Abstract summary: Gradient-boosted decision trees (GBDTs) are a powerful and widely-used class of models. We adapt influence-estimation methods designed for deep learning models to GBDTs. We find BoostIn is an efficient influence-estimation method for GBDTs that performs equally well or better than existing work.
• Score: 12.167833575680833
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Influence estimation analyzes how changes to the training data can lead to different model predictions; this analysis can help us better understand these predictions, the models making those predictions, and the data sets they're trained on. However, most influence-estimation techniques are designed for deep learning models with continuous parameters. Gradient-boosted decision trees (GBDTs) are a powerful and widely-used class of models; however, these models are black boxes with opaque decision-making processes. In the pursuit of better understanding GBDT predictions and generally improving these models, we adapt recent and popular influence-estimation methods designed for deep learning models to GBDTs. Specifically, we adapt representer-point methods and TracIn, denoting our new methods TREX and BoostIn, respectively; source code is available at https://github.com/jjbrophy47/tree_influence. We compare these methods to LeafInfluence and other baselines using 5 different evaluation measures on 22 real-world data sets with 4 popular GBDT implementations. These experiments give us a comprehensive overview of how different approaches to influence estimation work in GBDT models. We find BoostIn is an efficient influence-estimation method for GBDTs that performs equally well or better than existing work while being four orders of magnitude faster. Our evaluation also suggests the gold-standard approach of leave-one-out (LOO) retraining consistently identifies the single-most influential training example but performs poorly at finding the most influential set of training examples for a given target prediction.

Related papers

• First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation [8.788531432978802]
  Training samples influence/impact Large Language Model (LLM) decision-making is essential for effectively interpreting model decisions.<n>Current training sample influence estimation methods (also known as influence functions) undertake this goal by utilizing information flow through the model.<n>However, owing to the large model sizes of today consisting of billions of parameters, these influence computations are often restricted to some subset of model layers.
  arXiv  Detail & Related papers  (2025-11-06T00:47:07Z)
• Nonparametric Data Attribution for Diffusion Models [57.820618036556084]
  Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs.<n>We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images.
  arXiv  Detail & Related papers  (2025-10-16T03:37:16Z)
• Daunce: Data Attribution through Uncertainty Estimation [7.809316632545256]
  Training data attribution methods aim to identify which training examples influence a model's predictions on specific test data most.<n> Gradient-based TDA methods rely on gradients and second-order information, limiting their applicability at scale.<n>We introduce Daunce - a simple yet effective data attribution approach through uncertainty estimation.
  arXiv  Detail & Related papers  (2025-05-29T08:08:38Z)
• Enhancing Training Data Attribution with Representational Optimization [57.61977909113113]
  Training data attribution methods aim to measure how training data impacts a model's predictions.<n>We propose AirRep, a representation-based approach that closes this gap by learning task-specific and model-aligned representations explicitly for TDA.<n>AirRep introduces two key innovations: a trainable encoder tuned for attribution quality, and an attention-based pooling mechanism that enables accurate estimation of group-wise influence.
  arXiv  Detail & Related papers  (2025-05-24T05:17:53Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Scalable Influence and Fact Tracing for Large Language Model Pretraining [14.598556308631018]
  Training data attribution (TDA) methods aim to attribute model outputs back to specific training examples. This paper refines existing gradient-based methods to work effectively at scale.
  arXiv  Detail & Related papers  (2024-10-22T20:39:21Z)
• Influence Functions for Scalable Data Attribution in Diffusion Models [52.92223039302037]
  Diffusion models have led to significant advancements in generative modelling. Yet their widespread adoption poses challenges regarding data attribution and interpretability. In this paper, we aim to help address such challenges by developing an textitinfluence functions framework.
  arXiv  Detail & Related papers  (2024-10-17T17:59:02Z)
• Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration [74.09687562334682]
  We introduce a novel training data attribution method called Debias and Denoise Attribution (DDA) Our method significantly outperforms existing approaches, achieving an averaged AUC of 91.64%. DDA exhibits strong generality and scalability across various sources and different-scale models like LLaMA2, QWEN2, and Mistral.
  arXiv  Detail & Related papers  (2024-10-02T07:14:26Z)
• Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback [110.16220825629749]
  Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements.
  arXiv  Detail & Related papers  (2024-06-13T16:17:21Z)
• The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passes [30.30769701138665]
  We introduce and explore the Mirrored Influence Hypothesis, highlighting a reciprocal nature of influence between training and test data. Specifically, it suggests that evaluating the influence of training data on test predictions can be reformulated as an equivalent, yet inverse problem. We introduce a new method for estimating the influence of training data, which requires calculating gradients for specific test samples, paired with a forward pass for each training point.
  arXiv  Detail & Related papers  (2024-02-14T03:43:05Z)
• Bayesian Deep Learning for Remaining Useful Life Estimation via Stein Variational Gradient Descent [14.784809634505903]
  We show that Bayesian deep learning models trained via Stein variational gradient descent consistently outperform with respect to convergence speed and predictive performance. We propose a method to enhance performance based on the uncertainty information provided by the Bayesian models.
  arXiv  Detail & Related papers  (2024-02-02T02:21:06Z)
• A positive feedback method based on F-measure value for Salient Object Detection [1.9249287163937976]
  This paper proposes a positive feedback method based on F-measure value for salient object detection (SOD) Our proposed method takes an image to be detected and inputs it into several existing models to obtain their respective prediction maps. Experimental results on five publicly available datasets show that our proposed positive feedback method outperforms the latest 12 methods in five evaluation metrics for saliency map prediction.
  arXiv  Detail & Related papers  (2023-04-28T04:05:13Z)
• Guide the Learner: Controlling Product of Experts Debiasing Method Based on Token Attribution Similarities [17.082695183953486]
  A popular workaround is to train a robust model by re-weighting training examples based on a secondary biased model. Here, the underlying assumption is that the biased model resorts to shortcut features. We introduce a fine-tuning strategy that incorporates the similarity between the main and biased model attribution scores in a Product of Experts loss function.
  arXiv  Detail & Related papers  (2023-02-06T15:21:41Z)
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning [109.74041512359476]
  We study a number of design decisions for the predictive model in visual MBRL algorithms. We find that a range of design decisions that are often considered crucial, such as the use of latent spaces, have little effect on task performance. We show how this phenomenon is related to exploration and how some of the lower-scoring models on standard benchmarks will perform the same as the best-performing models when trained on the same training data.
  arXiv  Detail & Related papers  (2020-12-08T18:03:21Z)
• Efficient Estimation of Influence of a Training Instance [56.29080605123304]
  We propose an efficient method for estimating the influence of a training instance on a neural network model. Our method is inspired by dropout, which zero-masks a sub-network and prevents the sub-network from learning each training instance. We demonstrate that the proposed method can capture training influences, enhance the interpretability of error predictions, and cleanse the training dataset for improving generalization.
  arXiv  Detail & Related papers  (2020-12-08T04:31:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.