Quality at the Tail of Machine Learning Inference

• URL: http://arxiv.org/abs/2212.13925v3
• Date: Mon, 26 Feb 2024 15:15:14 GMT
• Title: Quality at the Tail of Machine Learning Inference
• Abstract summary: The study unveils a counterintuitive revelation: deep learning inference quality exhibits fluctuations due to inference time. To depict this phenomenon, the authors coin a new term, "tail quality," providing a more comprehensive evaluation. The research proposes an initial evaluation framework to analyze factors affecting quality fluctuations.
• Score: 6.463446901942362
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning inference should be subject to stringent inference time constraints while ensuring high inference quality, especially in safety-critical (e.g., autonomous driving) and mission-critical (e.g., emotion recognition) contexts. Neglecting either aspect can lead to severe consequences, such as loss of life and property damage. Many studies lack a comprehensive consideration of these metrics, leading to incomplete or misleading evaluations. The study unveils a counterintuitive revelation: deep learning inference quality exhibits fluctuations due to inference time. To depict this phenomenon, the authors coin a new term, "tail quality," providing a more comprehensive evaluation, and overcoming conventional metric limitations. Moreover, the research proposes an initial evaluation framework to analyze factors affecting quality fluctuations, facilitating the prediction of the potential distribution of inference quality. The effectiveness of the evaluation framework is validated through experiments conducted on deep learning models for three different tasks across four systems.

Related papers

• Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning [55.99621402404222]
  Reinforcement learning with verifiable rewards without human-annotated data, often referred to as zero RL, has emerged as a powerful paradigm for eliciting chain-of-thought reasoning.<n>We aim to elicit high-quality reasoning behaviors from the model.<n>We find that naive scaling often suffers from poor readability, token redundancy, and a lack of adaptive reasoning depth.
  arXiv  Detail & Related papers  (2026-07-14T06:14:55Z)
• Evaluation Pitfalls and Challenges in Multimedia Event Extraction [17.9089265435157]
  We present the first systematic analysis of evaluation pitfalls in multimedia event extraction.<n>We identify three major sources of issues: inconsistent data processing, inconsistent task assumptions, and overly relaxed evaluation settings.
  arXiv  Detail & Related papers  (2026-06-25T09:05:13Z)
• How Uncertain Is the Grade? A Benchmark of Uncertainty Metrics for LLM-Based Automatic Assessment [30.331175047465408]
  The rapid rise of large language models (LLMs) is reshaping the landscape of automatic assessment in education.<n> Output uncertainty is an inescapable challenge in automatic assessment.<n>Unreliable or poorly calibrated uncertainty estimates can lead to unstable downstream interventions.
  arXiv  Detail & Related papers  (2026-02-17T21:46:52Z)
• Bridging the Perceptual - Statistical Gap in Dysarthria Assessment: Why Machine Learning Still Falls Short [3.4181221698258066]
  Automated dysarthria detection and severity assessment from speech have attracted significant research attention due to their potential clinical impact.<n>Despite rapid progress in acoustic modeling and deep learning, models still fall short of human expert performance.<n>This manuscript provides a comprehensive analysis of the reasons behind this gap, emphasizing a conceptual divergence we term the perceptual-statistical gap''
  arXiv  Detail & Related papers  (2025-10-25T09:44:31Z)
• Failure Risk Prediction in a MOOC: A Multivariate Time Series Analysis Approach [0.7087237546722617]
  This work compares time series classification methods to identify at-risk learners at different stages of the course.<n>Preliminary results show that the evaluated approaches are promising for predicting learner failure in MOOCs.
  arXiv  Detail & Related papers  (2025-07-17T12:22:10Z)
• Existing Large Language Model Unlearning Evaluations Are Inconclusive [105.55899615056573]
  We show that some evaluations introduce substantial new information into the model, potentially masking true unlearning performance.<n>We demonstrate that evaluation outcomes vary significantly across tasks, undermining the generalizability of current evaluation routines.<n>We propose two principles for future unlearning evaluations: minimal information injection and downstream task awareness.
  arXiv  Detail & Related papers  (2025-05-31T19:43:00Z)
• Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Tasks [18.613353004764885]
  This study reveals novel insights into the limitations of existing methods.<n>We propose a refined set of evaluation standards, including inspection of reasoning paths, diversified assessment metrics, and comparative analyses with human performance.
  arXiv  Detail & Related papers  (2025-05-28T15:17:34Z)
• Class-Dependent Perturbation Effects in Evaluating Time Series Attributions [5.136283512042341]
  We show previously overlooked class-dependent effects in feature attribution metrics. Our analysis suggests that perturbation-based evaluation may reflect specific model behaviors rather than intrinsic attribution quality. We propose an evaluation framework with a class-aware penalty term to help assess and account for these effects.
  arXiv  Detail & Related papers  (2025-02-24T10:22:03Z)
• SoK: What Makes Private Learning Unfair? [8.37060553485295]
  We discuss the factors that contribute to the disparate effect of training models with differential privacy guarantees. We find that factors related to the training dataset and the underlying distribution play a decisive role in the occurrence of disparate impact.
  arXiv  Detail & Related papers  (2025-01-24T11:28:27Z)
• Self-Distilled Disentangled Learning for Counterfactual Prediction [49.84163147971955]
  We propose the Self-Distilled Disentanglement framework, known as $SD2$. Grounded in information theory, it ensures theoretically sound independent disentangled representations without intricate mutual information estimator designs. Our experiments, conducted on both synthetic and real-world datasets, confirm the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-06-09T16:58:19Z)
• A Theoretical and Practical Framework for Evaluating Uncertainty Calibration in Object Detection [1.8843687952462744]
  This work presents a novel theoretical and practical framework to evaluate object detection systems in the context of uncertainty calibration. The robustness of the proposed uncertainty calibration metrics is shown through a series of representative experiments.
  arXiv  Detail & Related papers  (2023-09-01T14:02:44Z)
• A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification [0.491574468325115]
  We present a large-scale empirical study for the first time enabling benchmarking confidence scoring functions. The revelation of a simple softmax response baseline as the overall best performing method underlines the drastic shortcomings of current evaluation.
  arXiv  Detail & Related papers  (2022-11-28T12:25:27Z)
• Fairness Increases Adversarial Vulnerability [50.90773979394264]
  This paper shows the existence of a dichotomy between fairness and robustness, and analyzes when achieving fairness decreases the model robustness to adversarial samples. Experiments on non-linear models and different architectures validate the theoretical findings in multiple vision domains. The paper proposes a simple, yet effective, solution to construct models achieving good tradeoffs between fairness and robustness.
  arXiv  Detail & Related papers  (2022-11-21T19:55:35Z)
• Uncertainty-Driven Action Quality Assessment [67.20617610820857]
  We propose a novel probabilistic model, named Uncertainty-Driven AQA (UD-AQA), to capture the diversity among multiple judge scores. We generate the estimation of uncertainty for each prediction, which is employed to re-weight AQA regression loss. Our proposed method achieves competitive results on three benchmarks including the Olympic events MTL-AQA and FineDiving, and the surgical skill JIGSAWS datasets.
  arXiv  Detail & Related papers  (2022-07-29T07:21:15Z)
• Exploring the Trade-off between Plausibility, Change Intensity and Adversarial Power in Counterfactual Explanations using Multi-objective Optimization [73.89239820192894]
  We argue that automated counterfactual generation should regard several aspects of the produced adversarial instances. We present a novel framework for the generation of counterfactual examples.
  arXiv  Detail & Related papers  (2022-05-20T15:02:53Z)
• Evaluation Gaps in Machine Learning Practice [13.963766987258161]
  In practice, evaluations of machine learning models frequently focus on a narrow range of decontextualized predictive behaviours. We examine the evaluation gaps between the idealized breadth of evaluation concerns and the observed narrow focus of actual evaluations. By studying these properties, we demonstrate the machine learning discipline's implicit assumption of a range of commitments which have normative impacts.
  arXiv  Detail & Related papers  (2022-05-11T04:00:44Z)
• Empirical Estimates on Hand Manipulation are Recoverable: A Step Towards Individualized and Explainable Robotic Support in Everyday Activities [80.37857025201036]
  Key challenge for robotic systems is to figure out the behavior of another agent. Processing correct inferences is especially challenging when (confounding) factors are not controlled experimentally. We propose equipping robots with the necessary tools to conduct observational studies on people.
  arXiv  Detail & Related papers  (2022-01-27T22:15:56Z)
• SurvITE: Learning Heterogeneous Treatment Effects from Time-to-Event Data [83.50281440043241]
  We study the problem of inferring heterogeneous treatment effects from time-to-event data. We propose a novel deep learning method for treatment-specific hazard estimation based on balancing representations.
  arXiv  Detail & Related papers  (2021-10-26T20:13:17Z)
• Residual Error: a New Performance Measure for Adversarial Robustness [85.0371352689919]
  A major challenge that limits the wide-spread adoption of deep learning has been their fragility to adversarial attacks. This study presents the concept of residual error, a new performance measure for assessing the adversarial robustness of a deep neural network. Experimental results using the case of image classification demonstrate the effectiveness and efficacy of the proposed residual error metric.
  arXiv  Detail & Related papers  (2021-06-18T16:34:23Z)
• Benchmarking Adversarial Robustness [47.168521143464545]
  We establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks. Based on the evaluation results, we draw several important findings and provide insights for future research.
  arXiv  Detail & Related papers  (2019-12-26T12:37:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.