Stochastic Bottleneck: Rateless Auto-Encoder for Flexible Dimensionality Reduction

• URL: http://arxiv.org/abs/2005.02870v1
• Date: Wed, 6 May 2020 14:47:42 GMT
• Title: Stochastic Bottleneck: Rateless Auto-Encoder for Flexible Dimensionality Reduction
• Abstract summary: We propose a new concept of rateless auto-encoders (RL-AEs) that enable a flexible latent dimensionality. We experimentally demonstrate that the proposed RL-AEs can achieve variable dimensionality reduction while achieving low distortion.
• Score: 16.42183040840934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a new concept of rateless auto-encoders (RL-AEs) that enable a flexible latent dimensionality, which can be seamlessly adjusted for varying distortion and dimensionality requirements. In the proposed RL-AEs, instead of a deterministic bottleneck architecture, we use an over-complete representation that is stochastically regularized with weighted dropouts, in a manner analogous to sparse AE (SAE). Unlike SAEs, our RL-AEs employ monotonically increasing dropout rates across the latent representation nodes such that the latent variables become sorted by importance like in principal component analysis (PCA). This is motivated by the rateless property of conventional PCA, where the least important principal components can be discarded to realize variable rate dimensionality reduction that gracefully degrades the distortion. In contrast, since the latent variables of conventional AEs are equally important for data reconstruction, they cannot be simply discarded to further reduce the dimensionality after the AE model is trained. Our proposed stochastic bottleneck framework enables seamless rate adaptation with high reconstruction performance, without requiring predetermined latent dimensionality at training. We experimentally demonstrate that the proposed RL-AEs can achieve variable dimensionality reduction while achieving low distortion compared to conventional AEs.

Related papers

• Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generation [50.139984798361375]
  We propose a training-free framework that removes redundancy via latent discrepancy.<n> LD-Pruning substantially reduces inference while maintaining high generation quality, achieving up to 2.35x speedup on Infinity-8B.
  arXiv  Detail & Related papers  (2026-05-29T19:34:39Z)
• Entropic Auto-Encoding via Implicit Free-Energy Minimization [0.0]
  We introduce Entropic Autoencoders (EAEs), a framework in which reconstruction loss is the only explicit objective.<n>We demonstrate that EAEs mitigate posterior collapse by learning non-Gaussian, multimodal latent distributions.
  arXiv  Detail & Related papers  (2026-05-15T16:43:45Z)
• Adaptive Visual Autoregressive Acceleration via Dual-Linkage Entropy Analysis [50.48301331112126]
  We propose NOVA, a training-free token reduction acceleration framework for Visual AutoRegressive modeling.<n>NOVA adaptively determines the acceleration activation scale during inference by online identifying the inflection point of scale entropy growth.<n>Experiments and analyses validate NOVA as a simple yet effective training-free acceleration framework.
  arXiv  Detail & Related papers  (2026-02-01T17:29:42Z)
• Learning Ordered Representations in Latent Space for Intrinsic Dimension Estimation via Principal Component Autoencoder [10.509144950561103]
  Autoencoders have long been considered a nonlinear extension of Principal Component Analysis (PCA)<n>We propose a novel autoencoder framework that integrates non-uniform variance regularization with an isometric constraint.<n>This design serves as a natural generalization of PCA, enabling the model to preserve key advantages, such as ordered representations and variance retention.
  arXiv  Detail & Related papers  (2026-01-27T04:24:21Z)
• Uncertainty-Guided Selective Adaptation Enables Cross-Platform Predictive Fluorescence Microscopy [65.15943255667733]
  We introduce Subnetwork Image Translation ADDA with automatic depth selection (SIT-ADDA-Auto)<n>We show that adapting only the earliest convolutional layers, while freezing deeper layers, yields reliable transfer.<n>Our results provide a design rule for label-free adaptation in microscopy and a recipe for field settings; the code is publicly available.
  arXiv  Detail & Related papers  (2025-11-15T03:01:05Z)
• Rotation Equivariant Arbitrary-scale Image Super-Resolution [62.41329042683779]
  The arbitrary-scale image super-resolution (ASISR) aims to achieve arbitrary-scale high-resolution recoveries from a low-resolution input image.<n>We make efforts to construct a rotation equivariant ASISR method in this study.
  arXiv  Detail & Related papers  (2025-08-07T08:51:03Z)
• EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning [53.88000987041739]
  Class-Incremental Learning (CIL) aims to enable AI models to continuously learn from sequentially arriving data of different classes over time.<n>We propose the Elastic Knowledge Preservation and Compensation (EKPC) method, integrating Importance-aware importance Regularization (IPR) and Trainable Semantic Drift Compensation (TSDC) for CIL.
  arXiv  Detail & Related papers  (2025-06-14T05:19:58Z)
• Likelihood-Free Variational Autoencoders [21.092166159353702]
  Variational Autoencoders (VAEs) typically rely on a probabilistic decoder with a predefined likelihood, most commonly an isotropic Gaussian, to model the data conditional on latent variables. We propose EnVAE, a novel likelihood-free generative framework that has a deterministic decoder and employs the energy score--a proper scoring rule--to build the reconstruction loss. Our framework offers a general, scalable, and statistically principled alternative for flexible and nonparametric distribution learning in generative modeling.
  arXiv  Detail & Related papers  (2025-04-24T14:44:46Z)
• Effort: Efficient Orthogonal Modeling for Generalizable AI-Generated Image Detection [66.16595174895802]
  Existing AI-generated image (AIGI) detection methods often suffer from limited generalization performance. In this paper, we identify a crucial yet previously overlooked asymmetry phenomenon in AIGI detection.
  arXiv  Detail & Related papers  (2024-11-23T19:10:32Z)
• Enhancing Adversarial Transferability with Adversarial Weight Tuning [50.01825144613307]
  adversarial examples (AEs) mislead the model while appearing benign to human observers.<n>AWT is a data-free tuning method that combines gradient-based and model-based attack methods to enhance the transferability of AEs.
  arXiv  Detail & Related papers  (2024-08-18T13:31:26Z)
• ED-VAE: Entropy Decomposition of ELBO in Variational Autoencoders [0.850206009406913]
  Variational Autoencoders (VAEs) are constrained by the limitations of the Evidence Lower Bound (ELBO) formulation. This work introduces the Entropy Decomposed Variational Autoencoder (ED-VAE), a novel re-formulation of the ELBO that explicitly includes entropy and cross-entropy components.
  arXiv  Detail & Related papers  (2024-07-09T12:09:21Z)
• Fast Unsupervised Tensor Restoration via Low-rank Deconvolution [0.09208007322096533]
  Low-rank Deconvolution (LRD) has appeared as a new multi-dimensional representation model that enjoys important efficiency and flexibility properties. We ask ourselves if this analytical model can compete against Deep Learning (DL) frameworks like Deep Image Prior (DIP) or Blind-Spot Networks (BSN)
  arXiv  Detail & Related papers  (2024-06-15T16:04:49Z)
• Learning Point Spread Function Invertibility Assessment for Image Deconvolution [14.062542012968313]
  We propose a metric that employs a non-linear approach to learn the invertibility of an arbitrary PSF using a neural network. A lower discrepancy between the mapped PSF and a unit impulse indicates a higher likelihood of successful inversion by a DL network.
  arXiv  Detail & Related papers  (2024-05-25T20:00:27Z)
• ESSAformer: Efficient Transformer for Hyperspectral Image Super-resolution [76.7408734079706]
  Single hyperspectral image super-resolution (single-HSI-SR) aims to restore a high-resolution hyperspectral image from a low-resolution observation. We propose ESSAformer, an ESSA attention-embedded Transformer network for single-HSI-SR with an iterative refining structure.
  arXiv  Detail & Related papers  (2023-07-26T07:45:14Z)
• Exploiting Diffusion Prior for Real-World Image Super-Resolution [75.5898357277047]
  We present a novel approach to leverage prior knowledge encapsulated in pre-trained text-to-image diffusion models for blind super-resolution. By employing our time-aware encoder, we can achieve promising restoration results without altering the pre-trained synthesis model.
  arXiv  Detail & Related papers  (2023-05-11T17:55:25Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Effective Invertible Arbitrary Image Rescaling [77.46732646918936]
  Invertible Neural Networks (INN) are able to increase upscaling accuracy significantly by optimizing the downscaling and upscaling cycle jointly. A simple and effective invertible arbitrary rescaling network (IARN) is proposed to achieve arbitrary image rescaling by training only one model in this work. It is shown to achieve a state-of-the-art (SOTA) performance in bidirectional arbitrary rescaling without compromising perceptual quality in LR outputs.
  arXiv  Detail & Related papers  (2022-09-26T22:22:30Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE [51.09507030387935]
  Wasserstein autoencoder (WAE) shows that matching two distributions is equivalent to minimizing a simple autoencoder (AE) loss under the constraint that the latent space of this AE matches a pre-specified prior distribution. We propose to use the contrastive learning framework that has been shown to be effective for self-supervised representation learning, as a means to resolve this problem. We show that using the contrastive learning framework to optimize the WAE loss achieves faster convergence and more stable optimization compared with existing popular algorithms for WAE.
  arXiv  Detail & Related papers  (2021-10-19T22:55:47Z)
• Disentangling Generative Factors of Physical Fields Using Variational Autoencoders [0.0]
  This work explores the use of variational autoencoders (VAEs) for non-linear dimension reduction. A disentangled decomposition is interpretable and can be transferred to a variety of tasks including generative modeling.
  arXiv  Detail & Related papers  (2021-09-15T16:02:43Z)
• To Regularize or Not To Regularize? The Bias Variance Trade-off in Regularized AEs [10.611727286504994]
  We study the effect of the latent prior on the generation deterministic quality of AE models. We show that our model, called FlexAE, is the new state-of-the-art for the AE based generative models.
  arXiv  Detail & Related papers  (2020-06-10T14:00:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.