ArchRepair: Block-Level Architecture-Oriented Repairing for Deep Neural Networks

• URL: http://arxiv.org/abs/2111.13330v1
• Date: Fri, 26 Nov 2021 06:35:15 GMT
• Title: ArchRepair: Block-Level Architecture-Oriented Repairing for Deep Neural Networks
• Authors: Hua Qi, Zhijie Wang, Qing Guo, Jianlang Chen, Felix Juefei-Xu, Lei Ma, Jianjun Zhao
• Abstract summary: We propose a novel repairing direction for deep neural networks (DNNs) at the block level. We propose adversarial-aware spectrum analysis for vulnerable block localization. We also propose the architecture-oriented search-based repairing that relaxes the targeted block to a continuous repairing search space.
• Score: 13.661704974188872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Over the past few years, deep neural networks (DNNs) have achieved tremendous success and have been continuously applied in many application domains. However, during the practical deployment in the industrial tasks, DNNs are found to be erroneous-prone due to various reasons such as overfitting, lacking robustness to real-world corruptions during practical usage. To address these challenges, many recent attempts have been made to repair DNNs for version updates under practical operational contexts by updating weights (i.e., network parameters) through retraining, fine-tuning, or direct weight fixing at a neural level. In this work, as the first attempt, we initiate to repair DNNs by jointly optimizing the architecture and weights at a higher (i.e., block) level. We first perform empirical studies to investigate the limitation of whole network-level and layer-level repairing, which motivates us to explore a novel repairing direction for DNN repair at the block level. To this end, we first propose adversarial-aware spectrum analysis for vulnerable block localization that considers the neurons' status and weights' gradients in blocks during the forward and backward processes, which enables more accurate candidate block localization for repairing even under a few examples. Then, we further propose the architecture-oriented search-based repairing that relaxes the targeted block to a continuous repairing search space at higher deep feature levels. By jointly optimizing the architecture and weights in that space, we can identify a much better block architecture. We implement our proposed repairing techniques as a tool, named ArchRepair, and conduct extensive experiments to validate the proposed method. The results show that our method can not only repair but also enhance accuracy & robustness, outperforming the state-of-the-art DNN repair techniques.

Related papers

• Reusing Deep Learning Models: Challenges and Directions in Software Engineering [3.733306025181894]
  Deep neural networks (DNNs) achieve state-of-the-art performance in many areas. DNNs are expensive to develop, both in intellectual effort and computational costs. This vision paper describes challenges in current approaches to DNN re-use.
  arXiv  Detail & Related papers  (2024-04-25T15:42:10Z)
• Lightweight Diffusion Models with Distillation-Based Block Neural Architecture Search [55.41583104734349]
  We propose to automatically remove structural redundancy in diffusion models with our proposed Diffusion Distillation-based Block-wise Neural Architecture Search (NAS) Given a larger pretrained teacher, we leverage DiffNAS to search for the smallest architecture which can achieve on-par or even better performance than the teacher. Different from previous block-wise NAS methods, DiffNAS contains a block-wise local search strategy and a retraining strategy with a joint dynamic loss.
  arXiv  Detail & Related papers  (2023-11-08T12:56:59Z)
• Transferability of Convolutional Neural Networks in Stationary Learning Tasks [96.00428692404354]
  We introduce a novel framework for efficient training of convolutional neural networks (CNNs) for large-scale spatial problems. We show that a CNN trained on small windows of such signals achieves a nearly performance on much larger windows without retraining. Our results show that the CNN is able to tackle problems with many hundreds of agents after being trained with fewer than ten.
  arXiv  Detail & Related papers  (2023-07-21T13:51:45Z)
• Heterogeneous Continual Learning [88.53038822561197]
  We propose a novel framework to tackle the continual learning (CL) problem with changing network architectures. We build on top of the distillation family of techniques and modify it to a new setting where a weaker model takes the role of a teacher. We also propose Quick Deep Inversion (QDI) to recover prior task visual features to support knowledge transfer.
  arXiv  Detail & Related papers  (2023-06-14T15:54:42Z)
• Architecture-Preserving Provable Repair of Deep Neural Networks [2.4687962186994663]
  Deep neural networks (DNNs) are becoming increasingly important components of software, and are considered the state-of-the-art solution for a number of problems. This paper addresses the problem of architecture-preserving V-polytope provable repair of DNNs.
  arXiv  Detail & Related papers  (2023-04-07T06:36:41Z)
• Incremental Satisfiability Modulo Theory for Verification of Deep Neural Networks [22.015676101940077]
  We present an incremental satisfiability modulo theory (SMT) algorithm based on the Reluplex framework. We implement our algorithm as an incremental solver called DeepInc, and exerimental results show that DeepInc is more efficient in most cases.
  arXiv  Detail & Related papers  (2023-02-10T04:31:28Z)
• Measurement-Consistent Networks via a Deep Implicit Layer for Solving Inverse Problems [0.0]
  End-to-end deep neural networks (DNNs) have become state-of-the-art (SOTA) for solving inverse problems. These networks are sensitive to minor variations in the training pipeline and often fail to reconstruct small but important details. We propose a framework that transforms any DNN for inverse problems into a measurement-consistent one.
  arXiv  Detail & Related papers  (2022-11-06T17:05:04Z)
• Self-Denoising Neural Networks for Few Shot Learning [66.38505903102373]
  We present a new training scheme that adds noise at multiple stages of an existing neural architecture while simultaneously learning to be robust to this added noise. This architecture, which we call a Self-Denoising Neural Network (SDNN), can be applied easily to most modern convolutional neural architectures.
  arXiv  Detail & Related papers  (2021-10-26T03:28:36Z)
• Provable Repair of Deep Neural Networks [8.55884254206878]
  Deep Neural Networks (DNNs) have grown in popularity over the past decade and are now being used in safety-critical domains such as aircraft collision avoidance. This paper tackles the problem of correcting a DNN once unsafe behavior is found. We introduce the provable repair problem, which is the problem of repairing a network N to construct a new network N' that satisfies a given specification.
  arXiv  Detail & Related papers  (2021-04-09T15:03:53Z)
• Disturbance-immune Weight Sharing for Neural Architecture Search [96.93812980299428]
  We propose a disturbance-immune update strategy for model updating. We theoretically analyze the effectiveness of our strategy in alleviating the performance disturbance risk.
  arXiv  Detail & Related papers  (2020-03-29T17:54:49Z)
• Robust Pruning at Initialization [61.30574156442608]
  A growing need for smaller, energy-efficient, neural networks to be able to use machine learning applications on devices with limited computational resources. For Deep NNs, such procedures remain unsatisfactory as the resulting pruned networks can be difficult to train and, for instance, they do not prevent one layer from being fully pruned.
  arXiv  Detail & Related papers  (2020-02-19T17:09:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.