Measuring the Sensitivity of Classification Models with the Error Sensitivity Profile

• URL: http://arxiv.org/abs/2604.25765v1
• Date: Tue, 28 Apr 2026 15:28:45 GMT
• Title: Measuring the Sensitivity of Classification Models with the Error Sensitivity Profile
• Abstract summary: We propose the Error Sensitivity Profile (ESP) to quantifies the sensitivity of model performance to errors in a single feature or in multiple features.<n>To support the computation of this metric, an integrated suite of tools, called dirty, is created.
• Score: 1.1878820609988696
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The quality of training data is critical to the performance of machine learning models. In this paper, the Error Sensitivity Profile (ESP) is proposed. It quantifies the sensitivity of model performance to errors in a single feature or in multiple features. By leveraging ESP, data-cleaning efforts can be prioritized based on error types and features most likely to affect model performance. To support the computation of this metric, an integrated suite of tools, called \dirty, is created. We conduct an extensive experimental study on two widely used datasets using 14 classification models, revealing that performance degradation is not always predictable from simple correlations with the target variable.

Related papers

• Attribution-Guided Model Rectification of Unreliable Neural Network Behaviors [60.06461883533697]
  We leverage rank-one model editing to establish an attribution-guided model rectification framework.<n>We first distinguish our rectification setting from existing model editing, yielding a formulation that corrects unreliable behavior.<n>To target the primary source of misbehavior, we introduce an attribution-guided layer localization method.
  arXiv  Detail & Related papers  (2026-03-08T01:06:07Z)
• Efficient Network Traffic Feature Sets for IoT Intrusion Detection [0.0]
  This work evaluates the feature sets provided by a combination of different feature selection methods, namely Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio, in multiple IoT network datasets. The influence of the smaller feature sets on both the classification performance and the training time of ML models is compared, with the aim of increasing the computational efficiency of IoT intrusion detection.
  arXiv  Detail & Related papers  (2024-06-12T09:51:29Z)
• Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models [24.073260299592675]
  In this paper, we establish a bridge between identifying detrimental training samples via influence functions and outlier gradient detection.<n>We first validate the hypothesis of our proposed outlier gradient analysis approach on synthetic datasets.<n>We then demonstrate its effectiveness in detecting mislabeled samples in vision models and selecting data samples for improving performance of natural language processing transformer models.
  arXiv  Detail & Related papers  (2024-05-06T21:34:46Z)
• The Importance of Model Inspection for Better Understanding Performance Characteristics of Graph Neural Networks [15.569758991934934]
  We investigate the effect of modelling choices on the feature learning characteristics of graph neural networks applied to a brain shape classification task. We find substantial differences in the feature embeddings at different layers of the models.
  arXiv  Detail & Related papers  (2024-05-02T13:26:18Z)
• Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program [0.0]
  This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks. We show that errors can be predicted with moderate model performance (AUC=0.65-0.75) and that model performance generalizes well across applications. We demonstrate the usefulness of the model in the context of auditing, where prioritizing tasks with high predicted error probabilities considerably increases the amount of corrected annotation errors.
  arXiv  Detail & Related papers  (2023-10-08T21:21:19Z)
• Quality In / Quality Out: Data quality more relevant than model choice in anomaly detection with the UGR'16 [0.29998889086656577]
  We show that relatively minor modifications on a benchmark dataset cause significantly more impact on model performance than the specific ML technique considered.<n>We also show that the measured model performance is uncertain, as a result of labelling inaccuracies.
  arXiv  Detail & Related papers  (2023-05-31T12:03:12Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)
• FairIF: Boosting Fairness in Deep Learning via Influence Functions with Validation Set Sensitive Attributes [51.02407217197623]
  We propose a two-stage training algorithm named FAIRIF. It minimizes the loss over the reweighted data set where the sample weights are computed. We show that FAIRIF yields models with better fairness-utility trade-offs against various types of bias.
  arXiv  Detail & Related papers  (2022-01-15T05:14:48Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)
• Influence Functions in Deep Learning Are Fragile [52.31375893260445]
  influence functions approximate the effect of samples in test-time predictions. influence estimates are fairly accurate for shallow networks. Hessian regularization is important to get highquality influence estimates.
  arXiv  Detail & Related papers  (2020-06-25T18:25:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.