Uncertainty Propagation in Convolutional Neural Networks: Technical Report

• URL: http://arxiv.org/abs/2102.06064v1
• Date: Thu, 11 Feb 2021 15:17:46 GMT
• Title: Uncertainty Propagation in Convolutional Neural Networks: Technical Report
• Authors: Christos Tzelepis and Ioannis Patras
• Abstract summary: We study the problem of propagation of uncertainty through typical building blocks of a Convolutional Neural Network (CNN) These include layers that perform linear operations, as well as layers that act non-linearly on their input, such as the Rectified Linear Unit (ReLU)
• Score: 13.224071661974593
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this technical report we study the problem of propagation of uncertainty (in terms of variances of given uni-variate normal random variables) through typical building blocks of a Convolutional Neural Network (CNN). These include layers that perform linear operations, such as 2D convolutions, fully-connected, and average pooling layers, as well as layers that act non-linearly on their input, such as the Rectified Linear Unit (ReLU). Finally, we discuss the sigmoid function, for which we give approximations of its first- and second-order moments, as well as the binary cross-entropy loss function, for which we approximate its expected value under normal random inputs.

Related papers

• Benign Overfitting for Regression with Trained Two-Layer ReLU Networks [14.36840959836957]
  We study the least-square regression problem with a two-layer fully-connected neural network, with ReLU activation function, trained by gradient flow. Our first result is a generalization result, that requires no assumptions on the underlying regression function or the noise other than that they are bounded.
  arXiv  Detail & Related papers  (2024-10-08T16:54:23Z)
• Learning Linear Causal Representations from Interventions under General Nonlinear Mixing [52.66151568785088]
  We prove strong identifiability results given unknown single-node interventions without access to the intervention targets. This is the first instance of causal identifiability from non-paired interventions for deep neural network embeddings.
  arXiv  Detail & Related papers  (2023-06-04T02:32:12Z)
• Fast Convergence in Learning Two-Layer Neural Networks with Separable Data [37.908159361149835]
  We study normalized gradient descent on two-layer neural nets. We prove for exponentially-tailed losses that using normalized GD leads to linear rate of convergence of the training loss to the global optimum.
  arXiv  Detail & Related papers  (2023-05-22T20:30:10Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Exploring Linear Feature Disentanglement For Neural Networks [63.20827189693117]
  Non-linear activation functions, e.g., Sigmoid, ReLU, and Tanh, have achieved great success in neural networks (NNs) Due to the complex non-linear characteristic of samples, the objective of those activation functions is to project samples from their original feature space to a linear separable feature space. This phenomenon ignites our interest in exploring whether all features need to be transformed by all non-linear functions in current typical NNs.
  arXiv  Detail & Related papers  (2022-03-22T13:09:17Z)
• Phenomenology of Double Descent in Finite-Width Neural Networks [29.119232922018732]
  Double descent delineates the behaviour of models depending on the regime they belong to. We use influence functions to derive suitable expressions of the population loss and its lower bound. Building on our analysis, we investigate how the loss function affects double descent.
  arXiv  Detail & Related papers  (2022-03-14T17:39:49Z)
• Convolutional Filtering and Neural Networks with Non Commutative Algebras [153.20329791008095]
  We study the generalization of non commutative convolutional neural networks. We show that non commutative convolutional architectures can be stable to deformations on the space of operators.
  arXiv  Detail & Related papers  (2021-08-23T04:22:58Z)
• Non-asymptotic Excess Risk Bounds for Classification with Deep Convolutional Neural Networks [6.051520664893158]
  We consider the problem of binary classification with a class of general deep convolutional neural networks. We define the prefactors of the risk bounds in terms of the input data dimension and other model parameters. We show that the classification methods with CNNs can circumvent the curse of dimensionality.
  arXiv  Detail & Related papers  (2021-05-01T15:55:04Z)
• Going beyond p-convolutions to learn grayscale morphological operators [64.38361575778237]
  We present two new morphological layers based on the same principle as the p-convolutional layer. In this work, we present two new morphological layers based on the same principle as the p-convolutional layer.
  arXiv  Detail & Related papers  (2021-02-19T17:22:16Z)
• Probabilistic Numeric Convolutional Neural Networks [80.42120128330411]
  Continuous input signals like images and time series that are irregularly sampled or have missing values are challenging for existing deep learning methods. We propose Probabilistic Convolutional Neural Networks which represent features as Gaussian processes (GPs) We then define a convolutional layer as the evolution of a PDE defined on this GP, followed by a nonlinearity. In experiments we show that our approach yields a $3times$ reduction of error from the previous state of the art on the SuperPixel-MNIST dataset and competitive performance on the medical time2012 dataset PhysioNet.
  arXiv  Detail & Related papers  (2020-10-21T10:08:21Z)
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer [5.969858080492586]
  We consider one dimensional (shallow) ReLU neural networks in which weights are chosen randomly and only the terminal layer is trained. We show that for such networks L2-regularized regression corresponds in function space to regularizing the estimate's second derivative for fairly general loss functionals.
  arXiv  Detail & Related papers  (2019-11-07T13:48:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.