Distributional Autoencoders Know the Score

• URL: http://arxiv.org/abs/2502.11583v2
• Date: Wed, 09 Apr 2025 17:56:17 GMT
• Title: Distributional Autoencoders Know the Score
• Authors: Andrej Leban,
• Abstract summary: The Distributional Principal Autoencoder (DPA) combines distributionally correct reconstruction with principal components-like interpretability of the encodings.<n>We show that the level sets of the encoder orient themselves exactly with regard to the score of the data distribution.<n>We prove that if the data lies on a manifold that can be approximated by the encoder, the optimal encoder's components beyond the dimension of the manifold will carry absolutely no additional information about the data distribution.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work presents novel and desirable properties of a recently introduced class of autoencoders - the Distributional Principal Autoencoder (DPA) - which combines distributionally correct reconstruction with principal components-like interpretability of the encodings. First, we show formally that the level sets of the encoder orient themselves exactly with regard to the score of the data distribution. This both explains the method's often remarkable performance in disentangling the factors of variation of the data, as well as opens up possibilities of recovering its distribution while having access to samples only. In settings where the score itself has physical meaning - such as when the data obeys the Boltzmann distribution - we demonstrate that the method can recover scientifically important quantities such as the minimum free energy path. Second, we prove that if the data lies on a manifold that can be approximated by the encoder, the optimal encoder's components beyond the dimension of the manifold will carry absolutely no additional information about the data distribution. This promises potentially new ways of determining the number of relevant dimensions of the data. The results thus demonstrate that the DPA elegantly combines two often disparate goals of unsupervised learning: the learning of the data distribution and the learning of the intrinsic data dimensionality.

Related papers

• Watermarking Generative Categorical Data [9.087950471621653]
  Our method embeds secret signals by splitting the data distribution into two components and modifying one distribution based on a deterministic relationship with the other. To verify the watermark, we introduce an insertion inverse algorithm and detect its presence by measuring the total variation distance between the inverse-decoded data and the original distribution.
  arXiv  Detail & Related papers  (2024-11-16T21:57:45Z)
• Adaptive Learning of the Latent Space of Wasserstein Generative Adversarial Networks [7.958528596692594]
  We propose a novel framework called the latent Wasserstein GAN (LWGAN) It fuses the Wasserstein auto-encoder and the Wasserstein GAN so that the intrinsic dimension of the data manifold can be adaptively learned. We show that LWGAN is able to identify the correct intrinsic dimension under several scenarios.
  arXiv  Detail & Related papers  (2024-09-27T01:25:22Z)
• Learned Compression of Encoding Distributions [1.4732811715354455]
  entropy bottleneck is a common component used in many learned compression models. We propose a method that adapts the encoding distribution to match the latent data distribution for a specific input. Our method achieves a Bjontegaard-Delta (BD)-rate gain of -7.10% on the Kodak test dataset.
  arXiv  Detail & Related papers  (2024-06-18T21:05:51Z)
• Distributional Principal Autoencoders [2.519266955671697]
  Dimension reduction techniques usually lose information in the sense that reconstructed data are not identical to the original data. We propose Distributional Principal Autoencoder (DPA) that consists of an encoder that maps high-dimensional data to low-dimensional latent variables. For reconstructing data, the DPA decoder aims to match the conditional distribution of all data that are mapped to a certain latent value.
  arXiv  Detail & Related papers  (2024-04-21T12:52:04Z)
• Beyond the Known: Adversarial Autoencoders in Novelty Detection [2.7486022583843233]
  In novelty detection, the goal is to decide if a new data point should be categorized as an inlier or an outlier. We use a similar framework but with a lightweight deep network, and we adopt a probabilistic score with reconstruction error. Our results indicate that our approach is effective at learning the target class, and it outperforms recent state-of-the-art methods on several benchmark datasets.
  arXiv  Detail & Related papers  (2024-04-06T00:04:19Z)
• Fact Checking Beyond Training Set [64.88575826304024]
  We show that the retriever-reader suffers from performance deterioration when it is trained on labeled data from one domain and used in another domain. We propose an adversarial algorithm to make the retriever component robust against distribution shift. We then construct eight fact checking scenarios from these datasets, and compare our model to a set of strong baseline models.
  arXiv  Detail & Related papers  (2024-03-27T15:15:14Z)
• Symmetric Equilibrium Learning of VAEs [56.56929742714685]
  We view variational autoencoders (VAEs) as decoder-encoder pairs, which map distributions in the data space to distributions in the latent space and vice versa. We propose a Nash equilibrium learning approach, which is symmetric with respect to the encoder and decoder and allows learning VAEs in situations where both the data and the latent distributions are accessible only by sampling.
  arXiv  Detail & Related papers  (2023-07-19T10:27:34Z)
• Probabilistic Matching of Real and Generated Data Statistics in Generative Adversarial Networks [0.6906005491572401]
  We propose a method to ensure that the distributions of certain generated data statistics coincide with the respective distributions of the real data. We evaluate the method on a synthetic dataset and a real-world dataset and demonstrate improved performance of our approach.
  arXiv  Detail & Related papers  (2023-06-19T14:03:27Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Convergent autoencoder approximation of low bending and low distortion manifold embeddings [5.5711773076846365]
  We propose and analyze a novel regularization for learning the encoder component of an autoencoder. The loss functional is computed via Monte Carlo integration with different sampling strategies for pairs of points on the input manifold. Our main theorem identifies a loss functional of the embedding map as the $Gamma$-limit of the sampling-dependent loss functionals.
  arXiv  Detail & Related papers  (2022-08-22T10:31:31Z)
• Discrete Key-Value Bottleneck [95.61236311369821]
  Deep neural networks perform well on classification tasks where data streams are i.i.d. and labeled data is abundant. One powerful approach that has addressed this challenge involves pre-training of large encoders on volumes of readily available data, followed by task-specific tuning. Given a new task, however, updating the weights of these encoders is challenging as a large number of weights needs to be fine-tuned, and as a result, they forget information about the previous tasks. We propose a model architecture to address this issue, building upon a discrete bottleneck containing pairs of separate and learnable key-value codes.
  arXiv  Detail & Related papers  (2022-07-22T17:52:30Z)
• Self-Conditioned Generative Adversarial Networks for Image Editing [61.50205580051405]
  Generative Adversarial Networks (GANs) are susceptible to bias, learned from either the unbalanced data, or through mode collapse. We argue that this bias is responsible not only for fairness concerns, but that it plays a key role in the collapse of latent-traversal editing methods when deviating away from the distribution's core.
  arXiv  Detail & Related papers  (2022-02-08T18:08:24Z)
• Neural Distributed Source Coding [59.630059301226474]
  We present a framework for lossy DSC that is agnostic to the correlation structure and can scale to high dimensions. We evaluate our method on multiple datasets and show that our method can handle complex correlations and state-of-the-art PSNR.
  arXiv  Detail & Related papers  (2021-06-05T04:50:43Z)
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components [96.42805872177067]
  We propose a self-supervised representation learning method for sequence data, based on the intuition that useful representations of sequence data should exhibit a simple structure in the latent space. We encourage this latent structure by maximizing an estimate of predictive information of latent feature sequences, which is the mutual information between past and future windows at each time step. We demonstrate that our method recovers the latent space of noisy dynamical systems, extracts predictive features for forecasting tasks, and improves automatic speech recognition when used to pretrain the encoder on large amounts of unlabeled data.
  arXiv  Detail & Related papers  (2020-10-07T03:34:01Z)
• Graph Embedding with Data Uncertainty [113.39838145450007]
  spectral-based subspace learning is a common data preprocessing step in many machine learning pipelines. Most subspace learning methods do not take into consideration possible measurement inaccuracies or artifacts that can lead to data with high uncertainty.
  arXiv  Detail & Related papers  (2020-09-01T15:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.