Testing Feedforward Neural Networks Training Programs

• URL: http://arxiv.org/abs/2204.00694v1
• Date: Fri, 1 Apr 2022 20:49:14 GMT
• Title: Testing Feedforward Neural Networks Training Programs
• Authors: Houssem Ben Braiek and Foutse Khomh
• Abstract summary: Multiple testing techniques are proposed to generate test cases that can expose inconsistencies in the behavior of Deep Neural Networks. These techniques assume implicitly that the training program is bug-free and appropriately configured. We propose TheDeepChecker, an end-to-end property-based debug approach for DNN training programs.
• Score: 13.249453757295083
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Nowadays, we are witnessing an increasing effort to improve the performance and trustworthiness of Deep Neural Networks (DNNs), with the aim to enable their adoption in safety critical systems such as self-driving cars. Multiple testing techniques are proposed to generate test cases that can expose inconsistencies in the behavior of DNN models. These techniques assume implicitly that the training program is bug-free and appropriately configured. However, satisfying this assumption for a novel problem requires significant engineering work to prepare the data, design the DNN, implement the training program, and tune the hyperparameters in order to produce the model for which current automated test data generators search for corner-case behaviors. All these model training steps can be error-prone. Therefore, it is crucial to detect and correct errors throughout all the engineering steps of DNN-based software systems and not only on the resulting DNN model. In this paper, we gather a catalog of training issues and based on their symptoms and their effects on the behavior of the training program, we propose practical verification routines to detect the aforementioned issues, automatically, by continuously validating that some important properties of the learning dynamics hold during the training. Then, we design, TheDeepChecker, an end-to-end property-based debugging approach for DNN training programs. We assess the effectiveness of TheDeepChecker on synthetic and real-world buggy DL programs and compare it with Amazon SageMaker Debugger (SMD). Results show that TheDeepChecker's on-execution validation of DNN-based program's properties succeeds in revealing several coding bugs and system misconfigurations, early on and at a low cost. Moreover, TheDeepChecker outperforms the SMD's offline rules verification on training logs in terms of detection accuracy and DL bugs coverage.

Related papers

• Search-based DNN Testing and Retraining with GAN-enhanced Simulations [2.362412515574206]
  In safety-critical systems, Deep Neural Networks (DNNs) are becoming a key component for computer vision tasks. We propose to combine meta-heuristic search, used to explore the input space using simulators, with Generative Adversarial Networks (GANs) to transform the data generated by simulators into realistic input images.
  arXiv  Detail & Related papers  (2024-06-19T09:05:16Z)
• Control-Theoretic Techniques for Online Adaptation of Deep Neural Networks in Dynamical Systems [0.0]
  Deep neural networks (DNNs) are currently the primary tool in modern artificial intelligence, machine learning, and data science. In many applications, DNNs are trained offline, through supervised learning or reinforcement learning, and deployed online for inference. We propose using techniques from control theory to update DNN parameters online.
  arXiv  Detail & Related papers  (2024-02-01T16:51:11Z)
• Cal-DETR: Calibrated Detection Transformer [67.75361289429013]
  We propose a mechanism for calibrated detection transformers (Cal-DETR), particularly for Deformable-DETR, UP-DETR and DINO. We develop an uncertainty-guided logit modulation mechanism that leverages the uncertainty to modulate the class logits. Results corroborate the effectiveness of Cal-DETR against the competing train-time methods in calibrating both in-domain and out-domain detections.
  arXiv  Detail & Related papers  (2023-11-06T22:13:10Z)
• DAD++: Improved Data-free Test Time Adversarial Defense [12.606555446261668]
  We propose a test time Data-free Adversarial Defense (DAD) containing detection and correction frameworks. We conduct a wide range of experiments and ablations on several datasets and network architectures to show the efficacy of our proposed approach. Our DAD++ gives an impressive performance against various adversarial attacks with a minimal drop in clean accuracy.
  arXiv  Detail & Related papers  (2023-09-10T20:39:53Z)
• An Effective Data-Driven Approach for Localizing Deep Learning Faults [20.33411443073181]
  We propose a novel data-driven approach that leverages model features to learn problem patterns. Our methodology automatically links bug symptoms to their root causes, without the need for manually crafted mappings. Our results demonstrate that our technique can effectively detect and diagnose different bug types.
  arXiv  Detail & Related papers  (2023-07-18T03:28:39Z)
• Adversarial Learning Networks: Source-free Unsupervised Domain Incremental Learning [0.0]
  In a non-stationary environment, updating a DNN model requires parameter re-training or model fine-tuning. We propose an unsupervised source-free method to update DNN classification models. Unlike existing methods, our approach can update a DNN model incrementally for non-stationary source and target tasks without storing past training data.
  arXiv  Detail & Related papers  (2023-01-28T02:16:13Z)
• The #DNN-Verification Problem: Counting Unsafe Inputs for Deep Neural Networks [94.63547069706459]
  #DNN-Verification problem involves counting the number of input configurations of a DNN that result in a violation of a safety property. We propose a novel approach that returns the exact count of violations. We present experimental results on a set of safety-critical benchmarks.
  arXiv  Detail & Related papers  (2023-01-17T18:32:01Z)
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning [92.36705236706678]
  "CodeRL" is a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning. During inference, we introduce a new generation procedure with a critical sampling strategy. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives.
  arXiv  Detail & Related papers  (2022-07-05T02:42:15Z)
• Adaptive Anomaly Detection for Internet of Things in Hierarchical Edge Computing: A Contextual-Bandit Approach [81.5261621619557]
  We propose an adaptive anomaly detection scheme with hierarchical edge computing (HEC) We first construct multiple anomaly detection DNN models with increasing complexity, and associate each of them to a corresponding HEC layer. Then, we design an adaptive model selection scheme that is formulated as a contextual-bandit problem and solved by using a reinforcement learning policy network.
  arXiv  Detail & Related papers  (2021-08-09T08:45:47Z)
• Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks [55.98291376393561]
  Graph neural networks (GNNs) have emerged as a powerful tool for learning software engineering tasks. Recurrent neural networks (RNNs) are well-suited to long sequential chains of reasoning, but they do not naturally incorporate program structure. We introduce a novel GNN architecture, the Instruction Pointer Attention Graph Neural Networks (IPA-GNN), which improves systematic generalization on the task of learning to execute programs.
  arXiv  Detail & Related papers  (2020-10-23T19:12:30Z)
• Boosting Deep Neural Networks with Geometrical Prior Knowledge: A Survey [77.99182201815763]
  Deep Neural Networks (DNNs) achieve state-of-the-art results in many different problem settings. DNNs are often treated as black box systems, which complicates their evaluation and validation. One promising field, inspired by the success of convolutional neural networks (CNNs) in computer vision tasks, is to incorporate knowledge about symmetric geometrical transformations.
  arXiv  Detail & Related papers  (2020-06-30T14:56:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.