Lifted Regression/Reconstruction Networks

• URL: http://arxiv.org/abs/2005.03452v1
• Date: Thu, 7 May 2020 13:24:46 GMT
• Title: Lifted Regression/Reconstruction Networks
• Authors: Rasmus Kj{\ae}r H{\o}ier, Christopher Zach
• Abstract summary: We propose lifted regression/reconstruction networks (LRRNs) LRRNs combine lifted neural networks with a guaranteed Lipschitz continuity property for the output layer. We analyse and numerically demonstrate applications for unsupervised and supervised learning.
• Score: 17.89437720094451
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work we propose lifted regression/reconstruction networks (LRRNs), which combine lifted neural networks with a guaranteed Lipschitz continuity property for the output layer. Lifted neural networks explicitly optimize an energy model to infer the unit activations and therefore---in contrast to standard feed-forward neural networks---allow bidirectional feedback between layers. So far lifted neural networks have been modelled around standard feed-forward architectures. We propose to take further advantage of the feedback property by letting the layers simultaneously perform regression and reconstruction. The resulting lifted network architecture allows to control the desired amount of Lipschitz continuity, which is an important feature to obtain adversarially robust regression and classification methods. We analyse and numerically demonstrate applications for unsupervised and supervised learning.

Related papers

• Benign Overfitting for Regression with Trained Two-Layer ReLU Networks [14.36840959836957]
  We study the least-square regression problem with a two-layer fully-connected neural network, with ReLU activation function, trained by gradient flow. Our first result is a generalization result, that requires no assumptions on the underlying regression function or the noise other than that they are bounded.
  arXiv  Detail & Related papers  (2024-10-08T16:54:23Z)
• Generalization and Estimation Error Bounds for Model-based Neural Networks [78.88759757988761]
  We show that the generalization abilities of model-based networks for sparse recovery outperform those of regular ReLU networks. We derive practical design rules that allow to construct model-based networks with guaranteed high generalization.
  arXiv  Detail & Related papers  (2023-04-19T16:39:44Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Improved Convergence Guarantees for Shallow Neural Networks [91.3755431537592]
  We prove convergence of depth 2 neural networks, trained via gradient descent, to a global minimum. Our model has the following features: regression with quadratic loss function, fully connected feedforward architecture, RelU activations, Gaussian data instances, adversarial labels. They strongly suggest that, at least in our model, the convergence phenomenon extends well beyond the NTK regime''
  arXiv  Detail & Related papers  (2022-12-05T14:47:52Z)
• Clustering-Based Interpretation of Deep ReLU Network [17.234442722611803]
  We recognize that the non-linear behavior of the ReLU function gives rise to a natural clustering. We propose a method to increase the level of interpretability of a fully connected feedforward ReLU neural network.
  arXiv  Detail & Related papers  (2021-10-13T09:24:11Z)
• Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics [0.5284812806199193]
  We take inspiration from a popular framework in neuroscience: 'predictive coding' We show that implementing this strategy into two popular networks, VGG16 and EfficientNetB0, improves their robustness against various corruptions.
  arXiv  Detail & Related papers  (2021-06-04T22:48:13Z)
• Self-Reorganizing and Rejuvenating CNNs for Increasing Model Capacity Utilization [8.661269034961679]
  We propose a biologically inspired method for improving the computational resource utilization of neural networks. The proposed method utilizes the channel activations of a convolution layer in order to reorganize that layers parameters. The rejuvenated parameters learn different features to supplement those learned by the reorganized surviving parameters.
  arXiv  Detail & Related papers  (2021-02-13T06:19:45Z)
• Local Critic Training for Model-Parallel Learning of Deep Neural Networks [94.69202357137452]
  We propose a novel model-parallel learning method, called local critic training. We show that the proposed approach successfully decouples the update process of the layer groups for both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) We also show that trained networks by the proposed method can be used for structural optimization.
  arXiv  Detail & Related papers  (2021-02-03T09:30:45Z)
• Implicit recurrent networks: A novel approach to stationary input processing with recurrent neural networks in deep learning [0.0]
  In this work, we introduce and test a novel implementation of recurrent neural networks into deep learning. We provide an algorithm which implements the backpropagation algorithm on a implicit implementation of recurrent networks. A single-layer implicit recurrent network is able to solve the XOR problem, while a feed-forward network with monotonically increasing activation function fails at this task.
  arXiv  Detail & Related papers  (2020-10-20T18:55:32Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)
• Lipschitz Recurrent Neural Networks [100.72827570987992]
  We show that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks.
  arXiv  Detail & Related papers  (2020-06-22T08:44:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.