Learning Manifold Dimensions with Conditional Variational Autoencoders

• URL: http://arxiv.org/abs/2302.11756v2
• Date: Tue, 13 Jun 2023 18:50:18 GMT
• Title: Learning Manifold Dimensions with Conditional Variational Autoencoders
• Authors: Yijia Zheng, Tong He, Yixuan Qiu, David Wipf
• Abstract summary: variational autoencoder (VAE) and its conditional extension (CVAE) are capable of state-of-the-art results across multiple domains. We show that VAE global minima are indeed capable of recovering the correct manifold dimension. We then extend this result to more general CVAEs, demonstrating practical scenarios.
• Score: 22.539599695796014
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although the variational autoencoder (VAE) and its conditional extension (CVAE) are capable of state-of-the-art results across multiple domains, their precise behavior is still not fully understood, particularly in the context of data (like images) that lie on or near a low-dimensional manifold. For example, while prior work has suggested that the globally optimal VAE solution can learn the correct manifold dimension, a necessary (but not sufficient) condition for producing samples from the true data distribution, this has never been rigorously proven. Moreover, it remains unclear how such considerations would change when various types of conditioning variables are introduced, or when the data support is extended to a union of manifolds (e.g., as is likely the case for MNIST digits and related). In this work, we address these points by first proving that VAE global minima are indeed capable of recovering the correct manifold dimension. We then extend this result to more general CVAEs, demonstrating practical scenarios whereby the conditioning variables allow the model to adaptively learn manifolds of varying dimension across samples. Our analyses, which have practical implications for various CVAE design choices, are also supported by numerical results on both synthetic and real-world datasets.

Related papers

• Estimating Dataset Dimension via Singular Metrics under the Manifold Hypothesis: Application to Inverse Problems [0.6138671548064356]
  We propose a framework to deal with three key tasks: estimating the intrinsic dimension of the manifold, constructing appropriate local coordinates, and learning mappings between ambient and manifold spaces.<n>We focus on estimating the ID of datasets by analyzing the numerical rank of the VAE decoder pullback metric.<n>The estimated ID guides the construction of an atlas of local charts using a mixture of invertible VAEs, enabling accurate manifold parameterization and efficient inference.
  arXiv  Detail & Related papers  (2025-07-09T21:22:59Z)
• Sparse Autoencoders, Again? [15.48801130346124]
  We formalize underappreciated weaknesses with both canonical SAEs and variational autoencoders.<n>We prove that global minima of our proposed model recover certain forms of structured data spread across a union of manifold.<n>In general, we are able to exceed the performance of equivalent-capacity SAEs and VAEs.
  arXiv  Detail & Related papers  (2025-06-05T10:26:06Z)
• Adaptive Learning of the Latent Space of Wasserstein Generative Adversarial Networks [7.958528596692594]
  We propose a novel framework called the latent Wasserstein GAN (LWGAN) It fuses the Wasserstein auto-encoder and the Wasserstein GAN so that the intrinsic dimension of the data manifold can be adaptively learned. We show that LWGAN is able to identify the correct intrinsic dimension under several scenarios.
  arXiv  Detail & Related papers  (2024-09-27T01:25:22Z)
• A Framework for Feasible Counterfactual Exploration incorporating Causality, Sparsity and Density [1.1782896991259]
  The imminent need to interpret a Machine Learning model with counterfactual (CF) explanations has been notable in the research community. This work uses different benchmark datasets to examine whether CF examples can be generated after a small amount of changes to the original input.
  arXiv  Detail & Related papers  (2024-04-20T22:05:48Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• Convolutional Filtering on Sampled Manifolds [122.06927400759021]
  We show that convolutional filtering on a sampled manifold converges to continuous manifold filtering. Our findings are further demonstrated empirically on a problem of navigation control.
  arXiv  Detail & Related papers  (2022-11-20T19:09:50Z)
• PAC Generalization via Invariant Representations [41.02828564338047]
  We consider the notion of $epsilon$-approximate invariance in a finite sample setting. Inspired by PAC learning, we obtain finite-sample out-of-distribution generalization guarantees. Our results show bounds that do not scale in ambient dimension when intervention sites are restricted to lie in a constant size subset of in-degree bounded nodes.
  arXiv  Detail & Related papers  (2022-05-30T15:50:14Z)
• Measuring dissimilarity with diffeomorphism invariance [94.02751799024684]
  We introduce DID, a pairwise dissimilarity measure applicable to a wide range of data spaces. We prove that DID enjoys properties which make it relevant for theoretical study and practical use.
  arXiv  Detail & Related papers  (2022-02-11T13:51:30Z)
• Hard-label Manifolds: Unexpected Advantages of Query Efficiency for Finding On-manifold Adversarial Examples [67.23103682776049]
  Recent zeroth order hard-label attacks on image classification models have shown comparable performance to their first-order, gradient-level alternatives. It was recently shown in the gradient-level setting that regular adversarial examples leave the data manifold, while their on-manifold counterparts are in fact generalization errors. We propose an information-theoretic argument based on a noisy manifold distance oracle, which leaks manifold information through the adversary's gradient estimate.
  arXiv  Detail & Related papers  (2021-03-04T20:53:06Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)
• Fork or Fail: Cycle-Consistent Training with Many-to-One Mappings [67.11712279612583]
  Cycle-consistent training is widely used for learning a forward and inverse mapping between two domains of interest. We develop a conditional variational autoencoder (CVAE) approach that can be viewed as converting surjective mappings to implicit bijections. Our pipeline can capture such many-to-one mappings during cycle training while promoting graph-to-text diversity.
  arXiv  Detail & Related papers  (2020-12-14T10:59:59Z)
• Geometry-Aware Hamiltonian Variational Auto-Encoder [0.0]
  Variational auto-encoders (VAEs) have proven to be a well suited tool for performing dimensionality reduction by extracting latent variables lying in a potentially much smaller dimensional space than the data. However, such generative models may perform poorly when trained on small data sets which are abundant in many real-life fields such as medicine. We argue that such latent space modelling provides useful information about its underlying structure leading to far more meaningfuls, more realistic data-generation and more reliable clustering.
  arXiv  Detail & Related papers  (2020-10-22T08:26:46Z)
• Variational Autoencoder with Learned Latent Structure [4.41370484305827]
  We introduce the Variational Autoencoder with Learned Latent Structure (VAELLS) VAELLS incorporates a learnable manifold model into the latent space of a VAE. We validate our model on examples with known latent structure and also demonstrate its capabilities on a real-world dataset.
  arXiv  Detail & Related papers  (2020-06-18T14:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.