Interpolation between Residual and Non-Residual Networks
- URL: http://arxiv.org/abs/2006.05749v4
- Date: Sun, 16 Aug 2020 22:02:41 GMT
- Title: Interpolation between Residual and Non-Residual Networks
- Authors: Zonghan Yang, Yang Liu, Chenglong Bao, Zuoqiang Shi
- Abstract summary: We present a novel ODE model by adding a damping term.
It can be shown that the proposed model can recover both a ResNet and a CNN by adjusting an coefficient.
Experiments on a number of image classification benchmarks show that the proposed model substantially improves the accuracy of ResNet and ResNeXt.
- Score: 24.690238357686134
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Although ordinary differential equations (ODEs) provide insights for
designing network architectures, its relationship with the non-residual
convolutional neural networks (CNNs) is still unclear. In this paper, we
present a novel ODE model by adding a damping term. It can be shown that the
proposed model can recover both a ResNet and a CNN by adjusting an
interpolation coefficient. Therefore, the damped ODE model provides a unified
framework for the interpretation of residual and non-residual networks. The
Lyapunov analysis reveals better stability of the proposed model, and thus
yields robustness improvement of the learned networks. Experiments on a number
of image classification benchmarks show that the proposed model substantially
improves the accuracy of ResNet and ResNeXt over the perturbed inputs from both
stochastic noise and adversarial attack methods. Moreover, the loss landscape
analysis demonstrates the improved robustness of our method along the attack
direction.
Related papers
- GFN: A graph feedforward network for resolution-invariant reduced operator learning in multifidelity applications [0.0]
This work presents a novel resolution-invariant model order reduction strategy for multifidelity applications.
We base our architecture on a novel neural network layer developed in this work, the graph feedforward network.
We exploit the method's capability of training and testing on different mesh sizes in an autoencoder-based reduction strategy for parametrised partial differential equations.
arXiv Detail & Related papers (2024-06-05T18:31:37Z) - Comprehensive Analysis of Network Robustness Evaluation Based on Convolutional Neural Networks with Spatial Pyramid Pooling [4.366824280429597]
Connectivity robustness, a crucial aspect for understanding, optimizing, and repairing complex networks, has traditionally been evaluated through simulations.
We address these challenges by designing a convolutional neural networks (CNN) model with spatial pyramid pooling networks (SPP-net)
We show that the proposed CNN model consistently achieves accurate evaluations of both attack curves and robustness values across all removal scenarios.
arXiv Detail & Related papers (2023-08-10T09:54:22Z) - A New PHO-rmula for Improved Performance of Semi-Structured Networks [0.0]
We show that techniques to properly identify the contributions of the different model components in SSNs lead to suboptimal network estimation.
We propose a non-invasive post-hocization (PHO) that guarantees identifiability of model components and provides better estimation and prediction quality.
Our theoretical findings are supported by numerical experiments, a benchmark comparison as well as a real-world application to COVID-19 infections.
arXiv Detail & Related papers (2023-06-01T10:23:28Z) - From Environmental Sound Representation to Robustness of 2D CNN Models
Against Adversarial Attacks [82.21746840893658]
This paper investigates the impact of different standard environmental sound representations (spectrograms) on the recognition performance and adversarial attack robustness of a victim residual convolutional neural network.
We show that while the ResNet-18 model trained on DWT spectrograms achieves a high recognition accuracy, attacking this model is relatively more costly for the adversary.
arXiv Detail & Related papers (2022-04-14T15:14:08Z) - Robust lEarned Shrinkage-Thresholding (REST): Robust unrolling for
sparse recover [87.28082715343896]
We consider deep neural networks for solving inverse problems that are robust to forward model mis-specifications.
We design a new robust deep neural network architecture by applying algorithm unfolding techniques to a robust version of the underlying recovery problem.
The proposed REST network is shown to outperform state-of-the-art model-based and data-driven algorithms in both compressive sensing and radar imaging problems.
arXiv Detail & Related papers (2021-10-20T06:15:45Z) - Kernel-Based Smoothness Analysis of Residual Networks [85.20737467304994]
Residual networks (ResNets) stand out among these powerful modern architectures.
In this paper, we show another distinction between the two models, namely, a tendency of ResNets to promote smoothers than gradients.
arXiv Detail & Related papers (2020-09-21T16:32:04Z) - An Ode to an ODE [78.97367880223254]
We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the group O(d)
This nested system of two flows provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem.
arXiv Detail & Related papers (2020-06-19T22:05:19Z) - Iterative Network for Image Super-Resolution [69.07361550998318]
Single image super-resolution (SISR) has been greatly revitalized by the recent development of convolutional neural networks (CNN)
This paper provides a new insight on conventional SISR algorithm, and proposes a substantially different approach relying on the iterative optimization.
A novel iterative super-resolution network (ISRN) is proposed on top of the iterative optimization.
arXiv Detail & Related papers (2020-05-20T11:11:47Z) - Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness [97.67477497115163]
We use mode connectivity to study the adversarial robustness of deep neural networks.
Our experiments cover various types of adversarial attacks applied to different network architectures and datasets.
Our results suggest that mode connectivity offers a holistic tool and practical means for evaluating and improving adversarial robustness.
arXiv Detail & Related papers (2020-04-30T19:12:50Z) - On generalized residue network for deep learning of unknown dynamical
systems [3.350695583277162]
We present a general numerical approach for learning unknown dynamical systems using deep neural networks (DNNs)
Our method is built upon recent studies that identified the residue network (ResNet) as an effective neural network structure.
arXiv Detail & Related papers (2020-01-23T01:50:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.