Attributing Data for Sharpness-Aware Minimization

• URL: http://arxiv.org/abs/2507.04059v1
• Date: Sat, 05 Jul 2025 14:46:42 GMT
• Title: Attributing Data for Sharpness-Aware Minimization
• Authors: Chenyang Ren, Yifan Jia, Huanyi Xie, Zhaobin Xu, Tianxing Wei, Liangyu Wang, Lijie Hu, Di Wang,
• Abstract summary: Sharpness-aware Minimization (SAM) improves generalization in large-scale model training by linking loss geometry to generalization.<n>However, challenges such as mislabeled noisy data and privacy concerns have emerged as significant issues.<n>We develop two innovative data valuation methods for SAM, each offering unique benefits in different scenarios.
• Score: 4.924675851574611
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sharpness-aware Minimization (SAM) improves generalization in large-scale model training by linking loss landscape geometry to generalization. However, challenges such as mislabeled noisy data and privacy concerns have emerged as significant issues. Data attribution, which identifies the contributions of specific training samples, offers a promising solution. However, directly rendering existing data influence evaluation tools such as influence functions (IF) to SAM will be inapplicable or inaccurate as SAM utilizes an inner loop to find model perturbations that maximize loss, which the outer loop then minimizes, resulting in a doubled computational structure. Additionally, this bilevel structure complicates the modeling of data influence on the parameters. In this paper, based on the IF, we develop two innovative data valuation methods for SAM, each offering unique benefits in different scenarios: the Hessian-based IF and the Gradient Trajectory-based IF. The first one provides a comprehensive estimation of data influence using a closed-form measure that relies only on the trained model weights. In contrast, the other IF for SAM utilizes gradient trajectory information during training for more accurate and efficient data assessment. Extensive experiments demonstrate their effectiveness in data evaluation and parameter tuning, with applications in identifying mislabeled data, model editing, and enhancing interpretability.

Related papers

• Model State Arithmetic for Machine Unlearning [43.773053236733425]
  We propose a new algorithm, MSA, for estimating and undoing the influence of datapoints.<n>Our experimental results demonstrate that MSA consistently outperforms existing machine unlearning algorithms.
  arXiv  Detail & Related papers  (2025-06-26T02:16:16Z)
• CALF: A Conditionally Adaptive Loss Function to Mitigate Class-Imbalanced Segmentation [0.2902243522110345]
  Imbalanced datasets pose a challenge in training deep learning (DL) models for medical diagnostics.<n>We propose a novel, statistically driven, conditionally adaptive loss function (CALF) tailored to accommodate the conditions of imbalanced datasets in DL training.
  arXiv  Detail & Related papers  (2025-04-06T12:03:33Z)
• Towards Robust Universal Information Extraction: Benchmark, Evaluation, and Solution [66.11004226578771]
  Existing robust benchmark datasets have two key limitations.<n>They generate only a limited range of perturbations for a single Information Extraction (IE) task.<n>Considering the powerful generation capabilities of Large Language Models (LLMs), we introduce a new benchmark dataset for Robust UIE, called RUIE-Bench.<n>We show that training with only textbf15% of the data leads to an average textbf7.5% relative performance improvement across three IE tasks.
  arXiv  Detail & Related papers  (2025-03-05T05:39:29Z)
• DUPRE: Data Utility Prediction for Efficient Data Valuation [49.60564885180563]
  Cooperative game theory-based data valuation, such as Data Shapley, requires evaluating the data utility and retraining the ML model for multiple data subsets.<n>Our framework, textttDUPRE, takes an alternative yet complementary approach that reduces the cost per subset evaluation by predicting data utilities instead of evaluating them by model retraining.<n>Specifically, given the evaluated data utilities of some data subsets, textttDUPRE fits a emphGaussian process (GP) regression model to predict the utility of every other data subset.
  arXiv  Detail & Related papers  (2025-02-22T08:53:39Z)
• Explainability of Machine Learning Models under Missing Data [3.0485328005356136]
  Missing data is a prevalent issue that can significantly impair model performance and explainability.<n>This paper briefly summarizes the development of the field of missing data and investigates the effects of various imputation methods on SHAP.
  arXiv  Detail & Related papers  (2024-06-29T11:31:09Z)
• Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models [89.88010750772413]
  Synthetic data has been proposed as a solution to address the issue of high-quality data scarcity in the training of large language models (LLMs) Our work delves into these specific flaws associated with question-answer (Q-A) pairs, a prevalent type of synthetic data, and presents a method based on unlearning techniques to mitigate these flaws. Our work has yielded key insights into the effective use of synthetic data, aiming to promote more robust and efficient LLM training.
  arXiv  Detail & Related papers  (2024-06-18T08:38:59Z)
• Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models [36.05242956018461]
  In this paper, we establish a bridge between identifying detrimental training samples via influence functions and outlier gradient detection.<n>We first validate the hypothesis of our proposed outlier gradient analysis approach on synthetic datasets.<n>We then demonstrate its effectiveness in detecting mislabeled samples in vision models and selecting data samples for improving performance of natural language processing transformer models.
  arXiv  Detail & Related papers  (2024-05-06T21:34:46Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• An empirical study of the effect of background data size on the stability of SHapley Additive exPlanations (SHAP) for deep learning models [14.65535880059975]
  We show that SHAP values and variable rankings fluctuate when using different background datasets acquired from random sampling. Our results suggest that users should take into account how background data affects SHAP results, with improved SHAP stability as the background sample size increases.
  arXiv  Detail & Related papers  (2022-04-24T20:15:34Z)
• FairIF: Boosting Fairness in Deep Learning via Influence Functions with Validation Set Sensitive Attributes [51.02407217197623]
  We propose a two-stage training algorithm named FAIRIF. It minimizes the loss over the reweighted data set where the sample weights are computed. We show that FAIRIF yields models with better fairness-utility trade-offs against various types of bias.
  arXiv  Detail & Related papers  (2022-01-15T05:14:48Z)
• How Training Data Impacts Performance in Learning-based Control [67.7875109298865]
  This paper derives an analytical relationship between the density of the training data and the control performance. We formulate a quality measure for the data set, which we refer to as $rho$-gap. We show how the $rho$-gap can be applied to a feedback linearizing control law.
  arXiv  Detail & Related papers  (2020-05-25T12:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.