High-dimensional Asymptotics of Denoising Autoencoders

• URL: http://arxiv.org/abs/2305.11041v1
• Date: Thu, 18 May 2023 15:35:11 GMT
• Title: High-dimensional Asymptotics of Denoising Autoencoders
• Authors: Hugo Cui, Lenka Zdeborov\'a
• Abstract summary: We address the problem of denoising data from a Gaussian mixture using a two-layer non-linear autoencoder with tied weights and a skip connection. We provide closed-form expressions for the denoising mean-squared test error.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We address the problem of denoising data from a Gaussian mixture using a two-layer non-linear autoencoder with tied weights and a skip connection. We consider the high-dimensional limit where the number of training samples and the input dimension jointly tend to infinity while the number of hidden units remains bounded. We provide closed-form expressions for the denoising mean-squared test error. Building on this result, we quantitatively characterize the advantage of the considered architecture over the autoencoder without the skip connection that relates closely to principal component analysis. We further show that our results accurately capture the learning curves on a range of real data sets.

Related papers

• Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth [83.15263499262824]
  We prove that gradient descent converges to a solution that completely disregards the sparse structure of the input. We show how to improve upon Gaussian performance for the compression of sparse data by adding a denoising function to a shallow architecture. We validate our findings on image datasets, such as CIFAR-10 and MNIST.
  arXiv  Detail & Related papers  (2024-02-07T16:32:29Z)
• Gappy local conformal auto-encoders for heterogeneous data fusion: in praise of rigidity [6.1152340690876095]
  We propose an end-to-end computational pipeline in the form of a multiple-auto-encoder neural network architecture for this task. The inputs to the pipeline are several sets of partial observations, and the result is a globally consistent latent space, harmonizing (rigidifying, fusing) all measurements. We demonstrate the approach in a sequence of examples, starting with simple two-dimensional data sets and proceeding to a Wi-Fi localization problem.
  arXiv  Detail & Related papers  (2023-12-20T16:18:51Z)
• Fighting over-fitting with quantization for learning deep neural networks on noisy labels [7.09232719022402]
  We study the ability of compression methods to tackle both of these problems at once. We hypothesize that quantization-aware training, by restricting the expressivity of neural networks, behaves as a regularization.
  arXiv  Detail & Related papers  (2023-03-21T12:36:58Z)
• Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods [91.54785981649228]
  This paper focuses on non-linear two-layer autoencoders trained in the challenging proportional regime. Our results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods. For the special case of a sign activation function, our analysis establishes the fundamental limits for the lossy compression of Gaussian sources via (shallow) autoencoders.
  arXiv  Detail & Related papers  (2022-12-27T12:37:34Z)
• DynImp: Dynamic Imputation for Wearable Sensing Data Through Sensory and Temporal Relatedness [78.98998551326812]
  We argue that traditional methods have rarely made use of both times-series dynamics of the data as well as the relatedness of the features from different sensors. We propose a model, termed as DynImp, to handle different time point's missingness with nearest neighbors along feature axis. We show that the method can exploit the multi-modality features from related sensors and also learn from history time-series dynamics to reconstruct the data under extreme missingness.
  arXiv  Detail & Related papers  (2022-09-26T21:59:14Z)
• Deep Double Descent via Smooth Interpolation [2.141079906482723]
  We quantify sharpness of fit of training data by studying the loss landscape w.r.t. to the input variable locally to each training point. Our findings show that loss sharpness in the input space follows both model- and epoch-wise double descent, with worse peaks observed around noisy targets. While small interpolating models sharply fit both clean and noisy data, large interpolating models express a smooth loss landscape, in contrast to existing intuition.
  arXiv  Detail & Related papers  (2022-09-21T02:46:13Z)
• Estimating High Order Gradients of the Data Distribution by Denoising [81.24581325617552]
  First order derivative of a data density can be estimated efficiently by denoising score matching. We propose a method to directly estimate high order derivatives (scores) of a data density from samples.
  arXiv  Detail & Related papers  (2021-11-08T18:59:23Z)
• Removing Noise from Extracellular Neural Recordings Using Fully Convolutional Denoising Autoencoders [62.997667081978825]
  We propose a Fully Convolutional Denoising Autoencoder, which learns to produce a clean neuronal activity signal from a noisy multichannel input. The experimental results on simulated data show that our proposed method can improve significantly the quality of noise-corrupted neural signals.
  arXiv  Detail & Related papers  (2021-09-18T14:51:24Z)
• Unsupervised Scale-consistent Depth Learning from Video [131.3074342883371]
  We propose a monocular depth estimator SC-Depth, which requires only unlabelled videos for training. Thanks to the capability of scale-consistent prediction, we show that our monocular-trained deep networks are readily integrated into the ORB-SLAM2 system. The proposed hybrid Pseudo-RGBD SLAM shows compelling results in KITTI, and it generalizes well to the KAIST dataset without additional training.
  arXiv  Detail & Related papers  (2021-05-25T02:17:56Z)
• Reconstruction of Sparse Signals under Gaussian Noise and Saturation [1.9873949136858349]
  Most compressed sensing algorithms do not account for the effect of saturation in noisy compressed measurements. We propose a new data fidelity function which is based on ensuring a certain form of consistency between the signal and the saturated measurements.
  arXiv  Detail & Related papers  (2021-02-08T03:01:46Z)
• Correlation-aware Deep Generative Model for Unsupervised Anomaly Detection [9.578395294627057]
  Unsupervised anomaly detection aims to identify anomalous samples from highly complex and unstructured data. We propose a method of Correlation aware unsupervised Anomaly detection via Deep Gaussian Mixture Model (CADGMM) Experiments on real-world datasets demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2020-02-18T03:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.