Variational Rank Reduction Autoencoder

• URL: http://arxiv.org/abs/2505.09458v1
• Date: Wed, 14 May 2025 15:08:28 GMT
• Title: Variational Rank Reduction Autoencoder
• Authors: Jad Mounayer, Alicia Tierz, Jerome Tomezyk, Chady Ghnatios, Francisco Chinesta,
• Abstract summary: We present Variational Rank Reduction Autoencoders (VRRAEs) a model that leverages the advantages of both RRAEs and VAEs.<n>Our results include a synthetic dataset of a small size that showcases the robustness of VRRAEs against collapse, and three real-world datasets.
• Score: 1.3980986259786223
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Deterministic Rank Reduction Autoencoders (RRAEs) enforce by construction a regularization on the latent space by applying a truncated SVD. While this regularization makes Autoencoders more powerful, using them for generative purposes is counter-intuitive due to their deterministic nature. On the other hand, Variational Autoencoders (VAEs) are well known for their generative abilities by learning a probabilistic latent space. In this paper, we present Variational Rank Reduction Autoencoders (VRRAEs), a model that leverages the advantages of both RRAEs and VAEs. Our claims and results show that when carefully sampling the latent space of RRAEs and further regularizing with the Kullback-Leibler (KL) divergence (similarly to VAEs), VRRAEs outperform RRAEs and VAEs. Additionally, we show that the regularization induced by the SVD not only makes VRRAEs better generators than VAEs, but also reduces the possibility of posterior collapse. Our results include a synthetic dataset of a small size that showcases the robustness of VRRAEs against collapse, and three real-world datasets; the MNIST, CelebA, and CIFAR-10, over which VRRAEs are shown to outperform both VAEs and RRAEs on many random generation and interpolation tasks based on the FID score.

Related papers

• KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding [72.12756830560217]
  Large language models (LLMs) based on Transformer Decoders have become the preferred choice for conversational generative AI.<n>Despite the overall superiority of the Decoder architecture, the gradually increasing Key-Value cache during inference has emerged as a primary efficiency bottleneck.<n>By down-sampling the Key-Value vector dimensions into a latent space, we can significantly reduce the KV Cache footprint and improve inference speed.
  arXiv  Detail & Related papers  (2025-07-15T12:52:12Z)
• Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model [63.336123527432136]
  We introduce Bench2Drive-R, a generative framework that enables reactive closed-loop evaluation.<n>Unlike existing video generative models for autonomous driving, the proposed designs are tailored for interactive simulation.<n>We compare the generation quality of Bench2Drive-R with existing generative models and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-11T06:35:18Z)
• Are Transformers in Pre-trained LM A Good ASR Encoder? An Empirical Study [52.91899050612153]
  transformers within pre-trained language models (PLMs) when repurposed as encoders for Automatic Speech Recognition (ASR) Our findings reveal a notable improvement in Character Error Rate (CER) and Word Error Rate (WER) across diverse ASR tasks when transformers from pre-trained LMs are incorporated. This underscores the potential of leveraging the semantic prowess embedded within pre-trained transformers to advance ASR systems' capabilities.
  arXiv  Detail & Related papers  (2024-09-26T11:31:18Z)
• Rank Reduction Autoencoders [3.180674374101366]
  We introduce a new class of deterministic autoencoders, Rank Reduction Autoencoders (RRAEs)<n>In RRAEs, the bottleneck is defined by the rank of the latent matrix, thereby alleviating the dependence of the encoder/decoder architecture on the bottleneck size.<n>We empirically demonstrate that both RRAEs and aRRAEs are stable, scalable, and reliable.
  arXiv  Detail & Related papers  (2024-05-22T20:33:09Z)
• How to train your VAE [0.0]
  Variational Autoencoders (VAEs) have become a cornerstone in generative modeling and representation learning within machine learning. This paper explores interpreting the Kullback-Leibler (KL) Divergence, a critical component within the Evidence Lower Bound (ELBO) The proposed method redefines the ELBO with a mixture of Gaussians for the posterior probability, introduces a regularization term, and employs a PatchGAN discriminator to enhance texture realism.
  arXiv  Detail & Related papers  (2023-09-22T19:52:28Z)
• VQ-T: RNN Transducers using Vector-Quantized Prediction Network States [52.48566999668521]
  We propose to use vector-quantized long short-term memory units in the prediction network of RNN transducers. By training the discrete representation jointly with the ASR network, hypotheses can be actively merged for lattice generation. Our experiments on the Switchboard corpus show that the proposed VQ RNN transducers improve ASR performance over transducers with regular prediction networks.
  arXiv  Detail & Related papers  (2022-08-03T02:45:52Z)
• Embrace the Gap: VAEs Perform Independent Mechanism Analysis [36.686468842036305]
  We study nonlinear VAEs in the limit of near-deterministic decoders. We show that VAEs uncover the true latent factors when the data generating process satisfies the IMA assumption.
  arXiv  Detail & Related papers  (2022-06-06T08:19:19Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE [51.09507030387935]
  Wasserstein autoencoder (WAE) shows that matching two distributions is equivalent to minimizing a simple autoencoder (AE) loss under the constraint that the latent space of this AE matches a pre-specified prior distribution. We propose to use the contrastive learning framework that has been shown to be effective for self-supervised representation learning, as a means to resolve this problem. We show that using the contrastive learning framework to optimize the WAE loss achieves faster convergence and more stable optimization compared with existing popular algorithms for WAE.
  arXiv  Detail & Related papers  (2021-10-19T22:55:47Z)
• Statistical Regeneration Guarantees of the Wasserstein Autoencoder with Latent Space Consistency [14.07437185521097]
  We investigate the statistical properties of Wasserstein Autoencoder (WAE) We provide statistical guarantees that WAE achieves the target distribution in the latent space. This study hints at the class of distributions WAE can reconstruct after suffering a compression in the form of a latent law.
  arXiv  Detail & Related papers  (2021-10-08T09:26:54Z)
• DAE : Discriminatory Auto-Encoder for multivariate time-series anomaly detection in air transportation [68.8204255655161]
  We propose a novel anomaly detection model called Discriminatory Auto-Encoder (DAE) It uses the baseline of a regular LSTM-based auto-encoder but with several decoders, each getting data of a specific flight phase. Results show that the DAE achieves better results in both accuracy and speed of detection.
  arXiv  Detail & Related papers  (2021-09-08T14:07:55Z)
• Variational Auto-Encoder: not all failures are equal [0.0]
  We show how sharpness learning addresses the notorious VAE blurriness issue. The paper is backed upon experiments on artificial data, MNIST and CelebA, showing how sharpness learning addresses the notorious VAE blurriness issue.
  arXiv  Detail & Related papers  (2020-03-04T09:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.