DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs

• URL: http://arxiv.org/abs/2112.04036v1
• Date: Tue, 7 Dec 2021 23:15:23 GMT
• Title: DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs
• Authors: Mohammad Wardat, Breno Dantas Cruz, Wei Le, Hridesh Rajan
• Abstract summary: We propose DeepDiagnosis, a novel approach that localizes the faults, reports error symptoms and suggests fixes for DNN programs. DeepDiagnosis manifests the best capabilities of fault detection, bug localization, and symptoms identification when compared to other approaches.
• Score: 12.917211542949786
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Neural Networks (DNNs) are used in a wide variety of applications. However, as in any software application, DNN-based apps are afflicted with bugs. Previous work observed that DNN bug fix patterns are different from traditional bug fix patterns. Furthermore, those buggy models are non-trivial to diagnose and fix due to inexplicit errors with several options to fix them. To support developers in locating and fixing bugs, we propose DeepDiagnosis, a novel debugging approach that localizes the faults, reports error symptoms and suggests fixes for DNN programs. In the first phase, our technique monitors a training model, periodically checking for eight types of error conditions. Then, in case of problems, it reports messages containing sufficient information to perform actionable repairs to the model. In the evaluation, we thoroughly examine 444 models -53 real-world from GitHub and Stack Overflow, and 391 curated by AUTOTRAINER. DeepDiagnosis provides superior accuracy when compared to UMLUAT and DeepLocalize. Our technique is faster than AUTOTRAINER for fault localization. The results show that our approach can support additional types of models, while state-of-the-art was only able to handle classification ones. Our technique was able to report bugs that do not manifest as numerical errors during training. Also, it can provide actionable insights for fix whereas DeepLocalize can only report faults that lead to numerical errors during training. DeepDiagnosis manifests the best capabilities of fault detection, bug localization, and symptoms identification when compared to other approaches.

Related papers

• A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection [52.228708947607636]
  This paper introduces a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework for new methods. The benchmark includes multiple datasets from industrial and medical domains, implementing fifteen state-of-the-art methods and nine comprehensive metrics. We objectively reveal the strengths and weaknesses of different methods and provide insights into the challenges and future directions of multi-class visual anomaly detection.
  arXiv  Detail & Related papers  (2024-06-05T13:40:07Z)
• A Deep Dive into Large Language Models for Automated Bug Localization and Repair [12.756202755547024]
  Large language models (LLMs) have shown impressive effectiveness in various software engineering tasks, including automated program repair (APR) In this study, we take a deep dive into automated bug fixing utilizing LLMs. This methodological separation of bug localization and fixing using different LLMs enables effective integration of diverse contextual information. Toggle achieves the new state-of-the-art (SOTA) performance on the CodeXGLUE code refinement benchmark.
  arXiv  Detail & Related papers  (2024-04-17T17:48:18Z)
• Mutation-based Fault Localization of Deep Neural Networks [11.93979764176335]
  Deep neural networks (DNNs) are susceptible to bugs, just like other types of software systems. This paper revisits mutation-based fault localization in the context of DNN models and proposes a novel technique. Deepmufl detects 53/109 of the bugs by ranking the buggy layer in top-1 position.
  arXiv  Detail & Related papers  (2023-09-10T16:18:49Z)
• An Effective Data-Driven Approach for Localizing Deep Learning Faults [20.33411443073181]
  We propose a novel data-driven approach that leverages model features to learn problem patterns. Our methodology automatically links bug symptoms to their root causes, without the need for manually crafted mappings. Our results demonstrate that our technique can effectively detect and diagnose different bug types.
  arXiv  Detail & Related papers  (2023-07-18T03:28:39Z)
• WELL: Applying Bug Detectors to Bug Localization via Weakly Supervised Learning [37.09621161662761]
  This paper proposes a WEakly supervised bug LocaLization (WELL) method to train a bug localization model. With CodeBERT finetuned on the buggy-or-not binary labeled data, WELL can address bug localization in a weakly supervised manner.
  arXiv  Detail & Related papers  (2023-05-27T06:34:26Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• Fast and Accurate Error Simulation for CNNs against Soft Errors [64.54260986994163]
  We present a framework for the reliability analysis of Conal Neural Networks (CNNs) via an error simulation engine. These error models are defined based on the corruption patterns of the output of the CNN operators induced by faults. We show that our methodology achieves about 99% accuracy of the fault effects w.r.t. SASSIFI, and a speedup ranging from 44x up to 63x w.r.t.FI, that only implements a limited set of error models.
  arXiv  Detail & Related papers  (2022-06-04T19:45:02Z)
• DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs [15.081278640511998]
  DeepFD is a learning-based fault diagnosis and localization framework. It maps the fault localization task to a learning problem. It correctly diagnoses 52% faulty DL programs, compared with around half (27%) achieved by the best state-of-the-art works.
  arXiv  Detail & Related papers  (2022-05-04T08:15:56Z)
• DapStep: Deep Assignee Prediction for Stack Trace Error rePresentation [61.99379022383108]
  We propose new deep learning models to solve the bug triage problem. The models are based on a bidirectional recurrent neural network with attention and on a convolutional neural network. To improve the quality of ranking, we propose using additional information from version control system annotations.
  arXiv  Detail & Related papers  (2022-01-14T00:16:57Z)
• Black-Box Diagnosis and Calibration on GAN Intra-Mode Collapse: A Pilot Study [116.05514467222544]
  Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism. One concern raised is whether the state-of-the-art GAN's learned distribution still suffers from mode collapse. This paper explores to diagnose GAN intra-mode collapse and calibrate that, in a novel black-box setting.
  arXiv  Detail & Related papers  (2021-07-23T06:03:55Z)
• NADS: Neural Architecture Distribution Search for Uncertainty Awareness [79.18710225716791]
  Machine learning (ML) systems often encounter Out-of-Distribution (OoD) errors when dealing with testing data coming from a distribution different from training data. Existing OoD detection approaches are prone to errors and even sometimes assign higher likelihoods to OoD samples. We propose Neural Architecture Distribution Search (NADS) to identify common building blocks among all uncertainty-aware architectures.
  arXiv  Detail & Related papers  (2020-06-11T17:39:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.