HeunNet: Extending ResNet using Heun's Methods

• URL: http://arxiv.org/abs/2105.06168v2
• Date: Fri, 14 May 2021 14:50:53 GMT
• Title: HeunNet: Extending ResNet using Heun's Methods
• Authors: Mehrdad Maleki and Mansura Habiba and Barak A. Pearlmutter
• Abstract summary: HeunNet is a predictor-corrector variant of ResNet. Heun's method is more accurate than Euler's. HeunNet achieves high accuracy with low computational time.
• Score: 1.0071258008543083
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There is an analogy between the ResNet (Residual Network) architecture for deep neural networks and an Euler solver for an ODE. The transformation performed by each layer resembles an Euler step in solving an ODE. We consider the Heun Method, which involves a single predictor-corrector cycle, and complete the analogy, building a predictor-corrector variant of ResNet, which we call a HeunNet. Just as Heun's method is more accurate than Euler's, experiments show that HeunNet achieves high accuracy with low computational (both training and test) time compared to both vanilla recurrent neural networks and other ResNet variants.

Related papers

• Poisson Hyperplane Processes with Rectified Linear Units [2.15746572209356]
  We establish the connection between the Poisson hyperplane processes (PHP) and two-layer ReLU neural networks.<n>We show that the PHP with a Gaussian prior is an alternative probabilistic representation to a two-layer ReLU neural network.<n>Our numerical experiments demonstrate that our proposed method outperforms the classic two-layer ReLU neural network.
  arXiv  Detail & Related papers  (2026-01-09T07:12:50Z)
• LinSATNet: The Positive Linear Satisfiability Neural Networks [116.65291739666303]
  This paper studies how to introduce the popular positive linear satisfiability to neural networks. We propose the first differentiable satisfiability layer based on an extension of the classic Sinkhorn algorithm for jointly encoding multiple sets of marginal distributions.
  arXiv  Detail & Related papers  (2024-07-18T22:05:21Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• GIT-Net: Generalized Integral Transform for Operator Learning [58.13313857603536]
  This article introduces GIT-Net, a deep neural network architecture for approximating Partial Differential Equation (PDE) operators. GIT-Net harnesses the fact that differential operators commonly used for defining PDEs can often be represented parsimoniously when expressed in specialized functional bases. Numerical experiments demonstrate that GIT-Net is a competitive neural network operator, exhibiting small test errors and low evaluations across a range of PDE problems.
  arXiv  Detail & Related papers  (2023-12-05T03:03:54Z)
• Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth [92.25666446274188]
  sinusoidal neural networks with activations have been proposed as an alternative to networks with traditional activation functions. We first propose a simplified version of such sinusoidal neural networks, which allows both for easier practical implementation and simpler theoretical analysis. We then analyze the behavior of these networks from the neural tangent kernel perspective and demonstrate that their kernel approximates a low-pass filter with an adjustable bandwidth.
  arXiv  Detail & Related papers  (2022-11-26T07:41:48Z)
• Neural Basis Functions for Accelerating Solutions to High Mach Euler Equations [63.8376359764052]
  We propose an approach to solving partial differential equations (PDEs) using a set of neural networks. We regress a set of neural networks onto a reduced order Proper Orthogonal Decomposition (POD) basis. These networks are then used in combination with a branch network that ingests the parameters of the prescribed PDE to compute a reduced order approximation to the PDE.
  arXiv  Detail & Related papers  (2022-08-02T18:27:13Z)
• Momentum Residual Neural Networks [22.32840998053339]
  We propose to change the forward rule of a ResNet by adding a momentum term. MomentumNets can be used as a drop-in replacement for any existing ResNet block. We show that MomentumNets have the same accuracy as ResNets, while having a much smaller memory footprint.
  arXiv  Detail & Related papers  (2021-02-15T22:24:52Z)
• Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations [0.0]
  We consider utilizing a residual neural network (ResNet) to solve ordinary differential equations. We apply forward Euler, Runge-Kutta2 and Runge-Kutta4 finite difference methods to generate three sets of targets training the ResNet. The well trained ResNet behaves just as its counterpart of the corresponding one-step finite difference method.
  arXiv  Detail & Related papers  (2021-01-10T17:34:10Z)
• A Residual Solver and Its Unfolding Neural Network for Total Variation Regularized Models [5.9622541907827875]
  This paper proposes to solve the Total Variation regularized models by finding the residual between the input and the unknown optimal solution. We numerically confirm that the residual solver can reach the same global optimal solutions as the classical method on 500 natural images. Both the proposed algorithm and neural network are successfully applied on several problems to demonstrate their effectiveness and efficiency.
  arXiv  Detail & Related papers  (2020-09-08T01:44:34Z)
• Implicit Euler ODE Networks for Single-Image Dehazing [33.34490764631837]
  We propose an efficient end-to-end multi-level implicit network (MI-Net) for the single image dehazing problem. Our method outperforms existing methods and achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-07-13T15:27:33Z)
• A Model-driven Deep Neural Network for Single Image Rain Removal [52.787356046951494]
  We propose a model-driven deep neural network for the task, with fully interpretable network structures. Based on the convolutional dictionary learning mechanism for representing rain, we propose a novel single image deraining model. All the rain kernels and operators can be automatically extracted, faithfully characterizing the features of both rain and clean background layers.
  arXiv  Detail & Related papers  (2020-05-04T09:13:25Z)
• A game-theoretic approach for Generative Adversarial Networks [2.995087247817663]
  Generative adversarial networks (GANs) are a class of generative models, known for producing accurate samples. Main bottleneck for their implementation is that the neural networks are very hard to train. We propose a relaxed forward-backward algorithm for GANs. We prove that when the pseudogradient mapping of the game is monotone, we have convergence to an exact solution or in a neighbourhood of it.
  arXiv  Detail & Related papers  (2020-03-30T17:14:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.