Evaluation-Strategy Gap in Fault Diagnosis of Deep Learning Programs

• URL: http://arxiv.org/abs/2606.26492v1
• Date: Thu, 25 Jun 2026 00:52:22 GMT
• Title: Evaluation-Strategy Gap in Fault Diagnosis of Deep Learning Programs
• Abstract summary: Deep Learning (DL) programs can fail during training for many reasons.<n> Techniques for diagnosing such failures are commonly assessed using within-program cross-validation.<n>We investigate this gap using DynFault, a corpus of 5,542 fault-injected training traces from 38 real-world DL programs.
• Score: 0.8883733362171032
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Learning (DL) programs can fail during training for many reasons, and diagnosing the cause is a costly and time-consuming maintenance task. Techniques for diagnosing such failures are commonly assessed using within-program cross-validation, which may be inadequate for deployment settings involving previously unseen programs. It is therefore necessary to assess how performance differs across these settings and to identify the causes of any performance gap in established fault diagnosis techniques for DL. We investigate this gap using DynFault, a corpus of 5,542 fault-injected training traces from 38 real-world DL programs. We found a gap of 0.190 in balanced accuracy for existing fault diagnosis techniques between within-program evaluation and holding out whole programs. We also found the gap comes from program-level structure in the features, which led us to examine two runtime feature sets, curvature features and optimizer features, and their behavior on unseen programs. We found that curvature features are useful for instability detection on unseen programs, while optimizer and activation features help only on programs seen during training.

Related papers

• Can Large Language Models Generate Observability-Aware Code? [51.30941152293301]
  We present a systematic study of observability in agent-generated systems.<n>Our results reveal a consistent gap between diagnostic semantics at the source level and fault signals at runtime.<n>We introduce an observability-oriented skill, which can serve as a guidance to improve both diagnostic semantics and fault-signal exposure.
  arXiv  Detail & Related papers  (2026-07-07T03:20:52Z)
• FreeGAD: A Training-Free yet Effective Approach for Graph Anomaly Detection [54.576802512108685]
  Graph Anomaly Detection (GAD) aims to identify nodes that deviate from the majority within a graph.<n>Existing approaches often suffer from high deployment costs and poor scalability due to their complex and resource-intensive training processes.<n>We propose FreeGAD, a novel training-free yet effective GAD method.
  arXiv  Detail & Related papers  (2025-08-14T12:37:20Z)
• Specification-Guided Repair of Arithmetic Errors in Dafny Programs using LLMs [79.74676890436174]
  We present an APR tool for Dafny that uses formal specifications as oracles for fault localization and repair.<n>We localize faults through a series of steps, which include using Hoare logic to determine the state of each statement within the program.<n>Our tool achieves 89.6% fault localization coverage and GPT-4o mini yields the highest repair success rate of 74.18%.
  arXiv  Detail & Related papers  (2025-07-04T15:36:12Z)
• Training Language Models to Generate Quality Code with Program Analysis Feedback [66.0854002147103]
  Code generation with large language models (LLMs) is increasingly adopted in production but fails to ensure code quality.<n>We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-quality code.
  arXiv  Detail & Related papers  (2025-05-28T17:57:47Z)
• How Execution Features Relate to Failures: An Empirical Study and Diagnosis Approach [11.857060911501016]
  Fault localization aims to identify code regions likely responsible for failures.<n>Traditional techniques primarily correlate statement execution with failures.<n>We analyzed 17 execution features and assessed their correlation with failure outcomes.
  arXiv  Detail & Related papers  (2025-02-25T22:00:05Z)
• Improved Detection and Diagnosis of Faults in Deep Neural Networks Using Hierarchical and Explainable Classification [3.2623791881739033]
  We present DEFault -- a novel technique to detect and diagnose faults in Deep Neural Networks (DNN) programs.<n>Our approach achieves 94% recall in detecting real-world faulty DNN programs and 63% recall in diagnosing the root causes of the faults, demonstrating 3.92% - 11.54% higher performance than that of state-of-the-art techniques.
  arXiv  Detail & Related papers  (2025-01-22T00:55:09Z)
• Analysing software failure using runtime verification and LTL [0.0]
  A self-healing software system is designed to detect, diagnose, and automatically recover from faults or errors without human intervention. This research proposes a method using runtime verification to diagnose four types of errors at the component level.
  arXiv  Detail & Related papers  (2024-02-15T13:37:45Z)
• Weak Supervision Performance Evaluation via Partial Identification [46.73061437177238]
  Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels. We present a novel method to address this challenge by framing model evaluation as a partial identification problem. Our approach derives reliable bounds on key metrics without requiring labeled data, overcoming core limitations in current weak supervision evaluation techniques.
  arXiv  Detail & Related papers  (2023-12-07T07:15:11Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs [15.081278640511998]
  DeepFD is a learning-based fault diagnosis and localization framework. It maps the fault localization task to a learning problem. It correctly diagnoses 52% faulty DL programs, compared with around half (27%) achieved by the best state-of-the-art works.
  arXiv  Detail & Related papers  (2022-05-04T08:15:56Z)
• Estimating Structural Target Functions using Machine Learning and Influence Functions [103.47897241856603]
  We propose a new framework for statistical machine learning of target functions arising as identifiable functionals from statistical models. This framework is problem- and model-agnostic and can be used to estimate a broad variety of target parameters of interest in applied statistics. We put particular focus on so-called coarsening at random/doubly robust problems with partially unobserved information.
  arXiv  Detail & Related papers  (2020-08-14T16:48:29Z)
• Probabilistic Diagnostic Tests for Degradation Problems in Supervised Learning [0.0]
  Problems such as class imbalance, overlapping, small-disjuncts, noisy labels, and sparseness limit accuracy in classification algorithms. Probability diagnostic model based on identifying signs and symptoms of each problem is presented. Behavior and performance of several supervised algorithms are studied when training sets have such problems.
  arXiv  Detail & Related papers  (2020-04-06T20:32:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.