Early Time Classification with Accumulated Accuracy Gap Control

• URL: http://arxiv.org/abs/2402.00857v1
• Date: Thu, 1 Feb 2024 18:54:34 GMT
• Title: Early Time Classification with Accumulated Accuracy Gap Control
• Authors: Liran Ringel, Regev Cohen, Daniel Freedman, Michael Elad, Yaniv Romano
• Abstract summary: Early time classification algorithms aim to label a stream of features without processing the full input stream. We introduce a statistical framework that can be applied to any sequential classifier, formulating a calibrated stopping rule. We show that our proposed early stopping mechanism reduces up to 94% of timesteps used for classification while achieving rigorous accuracy gap control.
• Score: 34.77841988415891
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Early time classification algorithms aim to label a stream of features without processing the full input stream, while maintaining accuracy comparable to that achieved by applying the classifier to the entire input. In this paper, we introduce a statistical framework that can be applied to any sequential classifier, formulating a calibrated stopping rule. This data-driven rule attains finite-sample, distribution-free control of the accuracy gap between full and early-time classification. We start by presenting a novel method that builds on the Learn-then-Test calibration framework to control this gap marginally, on average over i.i.d. instances. As this algorithm tends to yield an excessively high accuracy gap for early halt times, our main contribution is the proposal of a framework that controls a stronger notion of error, where the accuracy gap is controlled conditionally on the accumulated halt times. Numerical experiments demonstrate the effectiveness, applicability, and usefulness of our method. We show that our proposed early stopping mechanism reduces up to 94% of timesteps used for classification while achieving rigorous accuracy gap control.

Related papers

• Acquiring Better Load Estimates by Combining Anomaly and Change Point Detection in Power Grid Time-series Measurements [0.49478969093606673]
  Our approach prioritizes interpretability while ensuring robust and generalizable performance on unseen data. Results indicate the clear wasted potential when filtering is not applied. Our methodology's interpretability makes it particularly suitable for critical infrastructure planning.
  arXiv  Detail & Related papers  (2024-05-25T10:15:51Z)
• An Adaptive Method for Weak Supervision with Drifting Data [11.035811912078216]
  We introduce an adaptive method with formal quality guarantees for weak supervision in a non-stationary setting. We focus on the non-stationary case, where the accuracy of the weak supervision sources can drift over time. Our algorithm does not require any assumptions on the magnitude of the drift, and it adapts based on the input.
  arXiv  Detail & Related papers  (2023-06-02T16:27:34Z)
• Context-Aware Selective Label Smoothing for Calibrating Sequence Recognition Model [16.7796720078021]
  We propose a Context-Aware Selective Label Smoothing (CASLS) method for calibrating sequential data. Results on sequence recognition tasks, including scene text recognition and speech recognition, demonstrate that our method can achieve the state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-03-13T09:27:52Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Adapting to Continuous Covariate Shift via Online Density Ratio Estimation [64.8027122329609]
  Dealing with distribution shifts is one of the central challenges for modern machine learning. We propose an online method that can appropriately reuse historical information. Our density ratio estimation method is proven to perform well by enjoying a dynamic regret bound.
  arXiv  Detail & Related papers  (2023-02-06T04:03:33Z)
• Conformal inference is (almost) free for neural networks trained with early stopping [1.2891210250935146]
  Early stopping based on hold-out data is a popular regularization technique designed to mitigate overfitting and increase the predictive accuracy of neural networks. This paper addresses the limitation with conformalized early stopping: a novel method that combines early stopping with conformal calibration while efficiently recycling the same hold-out data.
  arXiv  Detail & Related papers  (2023-01-27T06:43:07Z)
• Sample-dependent Adaptive Temperature Scaling for Improved Calibration [95.7477042886242]
  Post-hoc approach to compensate for neural networks being wrong is to perform temperature scaling. We propose to predict a different temperature value for each input, allowing us to adjust the mismatch between confidence and accuracy. We test our method on the ResNet50 and WideResNet28-10 architectures using the CIFAR10/100 and Tiny-ImageNet datasets.
  arXiv  Detail & Related papers  (2022-07-13T14:13:49Z)
• Investigation of Different Calibration Methods for Deep Speaker Embedding based Verification Systems [66.61691401921296]
  This paper presents an investigation over several methods of score calibration for deep speaker embedding extractors. An additional focus of this research is to estimate the impact of score normalization on the calibration performance of the system.
  arXiv  Detail & Related papers  (2022-03-28T21:22:22Z)
• Learning Prediction Intervals for Regression: Generalization and Calibration [12.576284277353606]
  We study the generation of prediction intervals in regression for uncertainty quantification. We use a general learning theory to characterize the optimality-feasibility tradeoff that encompasses Lipschitz continuity and VC-subgraph classes. We empirically demonstrate the strengths of our interval generation and calibration algorithms in terms of testing performances compared to existing benchmarks.
  arXiv  Detail & Related papers  (2021-02-26T17:55:30Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.