DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs

• URL: http://arxiv.org/abs/2205.01938v1
• Date: Wed, 4 May 2022 08:15:56 GMT
• Title: DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs
• Authors: Jialun Cao and Meiziniu Li and Xiao Chen and Ming Wen and Yongqiang Tian and Bo Wu and Shing-Chi Cheung
• Abstract summary: DeepFD is a learning-based fault diagnosis and localization framework. It maps the fault localization task to a learning problem. It correctly diagnoses 52% faulty DL programs, compared with around half (27%) achieved by the best state-of-the-art works.
• Score: 15.081278640511998
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Deep Learning (DL) systems are widely deployed for mission-critical applications, debugging such systems becomes essential. Most existing works identify and repair suspicious neurons on the trained Deep Neural Network (DNN), which, unfortunately, might be a detour. Specifically, several existing studies have reported that many unsatisfactory behaviors are actually originated from the faults residing in DL programs. Besides, locating faulty neurons is not actionable for developers, while locating the faulty statements in DL programs can provide developers with more useful information for debugging. Though a few recent studies were proposed to pinpoint the faulty statements in DL programs or the training settings (e.g. too large learning rate), they were mainly designed based on predefined rules, leading to many false alarms or false negatives, especially when the faults are beyond their capabilities. In view of these limitations, in this paper, we proposed DeepFD, a learning-based fault diagnosis and localization framework which maps the fault localization task to a learning problem. In particular, it infers the suspicious fault types via monitoring the runtime features extracted during DNN model training and then locates the diagnosed faults in DL programs. It overcomes the limitations by identifying the root causes of faults in DL programs instead of neurons and diagnosing the faults by a learning approach instead of a set of hard-coded rules. The evaluation exhibits the potential of DeepFD. It correctly diagnoses 52% faulty DL programs, compared with around half (27%) achieved by the best state-of-the-art works. Besides, for fault localization, DeepFD also outperforms the existing works, correctly locating 42% faulty programs, which almost doubles the best result (23%) achieved by the existing works.

Related papers

• VDebugger: Harnessing Execution Feedback for Debugging Visual Programs [103.61860743476933]
  We introduce V Debugger, a critic-refiner framework trained to localize and debug visual programs by tracking execution step by step. V Debugger identifies and corrects program errors leveraging detailed execution feedback, improving interpretability and accuracy. Evaluations on six datasets demonstrate V Debugger's effectiveness, showing performance improvements of up to 3.2% in downstream task accuracy.
  arXiv  Detail & Related papers  (2024-06-19T11:09:16Z)
• Mind the Error! Detection and Localization of Instruction Errors in Vision-and-Language Navigation [65.25839671641218]
  We propose a novel benchmark dataset that introduces various types of instruction errors considering potential human causes. We observe a noticeable performance drop (up to -25%) in Success Rate when evaluating the state-of-the-art VLN-CE methods on our benchmark. We also propose an effective method, based on a cross-modal transformer architecture, that achieves the best performance in error detection and localization.
  arXiv  Detail & Related papers  (2024-03-15T21:36:15Z)
• Demystifying Faulty Code with LLM: Step-by-Step Reasoning for Explainable Fault Localization [5.7821087202452]
  This study investigates the step-by-step reasoning for explainable fault localization. We created a dataset of faulty code files, along with explanations for 600 faulty lines. We found that for 22 out of the 30 randomly sampled cases, FuseFL generated correct explanations.
  arXiv  Detail & Related papers  (2024-03-15T17:47:20Z)
• An Effective Data-Driven Approach for Localizing Deep Learning Faults [20.33411443073181]
  We propose a novel data-driven approach that leverages model features to learn problem patterns. Our methodology automatically links bug symptoms to their root causes, without the need for manually crafted mappings. Our results demonstrate that our technique can effectively detect and diagnose different bug types.
  arXiv  Detail & Related papers  (2023-07-18T03:28:39Z)
• ISimDL: Importance Sampling-Driven Acceleration of Fault Injection Simulations for Evaluating the Robustness of Deep Learning [10.757663798809144]
  We propose ISimDL, a novel methodology that employs neuron sensitivity to generate importance sampling-based fault-scenarios. Our experiments show that the importance sampling provides up to 15x higher precision in selecting critical faults than the random uniform sampling, reaching such precision in less than 100 faults.
  arXiv  Detail & Related papers  (2023-03-14T16:15:28Z)
• DOMINO: Domain-aware Loss for Deep Learning Calibration [49.485186880996125]
  This paper proposes a novel domain-aware loss function to calibrate deep learning models. The proposed loss function applies a class-wise penalty based on the similarity between classes within a given target domain.
  arXiv  Detail & Related papers  (2023-02-10T09:47:46Z)
• Fault-Aware Neural Code Rankers [64.41888054066861]
  We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
  arXiv  Detail & Related papers  (2022-06-04T22:01:05Z)
• Fast and Accurate Error Simulation for CNNs against Soft Errors [64.54260986994163]
  We present a framework for the reliability analysis of Conal Neural Networks (CNNs) via an error simulation engine. These error models are defined based on the corruption patterns of the output of the CNN operators induced by faults. We show that our methodology achieves about 99% accuracy of the fault effects w.r.t. SASSIFI, and a speedup ranging from 44x up to 63x w.r.t.FI, that only implements a limited set of error models.
  arXiv  Detail & Related papers  (2022-06-04T19:45:02Z)
• Challenges in Migrating Imperative Deep Learning Programs to Graph Execution: An Empirical Study [4.415977307120617]
  We conduct a data-driven analysis of challenges -- and resultant bugs -- involved in writing reliable yet performant imperative DL code. We put forth several recommendations, best practices, and anti-patterns for effectively hybridizing imperative DL code.
  arXiv  Detail & Related papers  (2022-01-24T21:12:38Z)
• DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs [12.917211542949786]
  We propose DeepDiagnosis, a novel approach that localizes the faults, reports error symptoms and suggests fixes for DNN programs. DeepDiagnosis manifests the best capabilities of fault detection, bug localization, and symptoms identification when compared to other approaches.
  arXiv  Detail & Related papers  (2021-12-07T23:15:23Z)
• NADS: Neural Architecture Distribution Search for Uncertainty Awareness [79.18710225716791]
  Machine learning (ML) systems often encounter Out-of-Distribution (OoD) errors when dealing with testing data coming from a distribution different from training data. Existing OoD detection approaches are prone to errors and even sometimes assign higher likelihoods to OoD samples. We propose Neural Architecture Distribution Search (NADS) to identify common building blocks among all uncertainty-aware architectures.
  arXiv  Detail & Related papers  (2020-06-11T17:39:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.