Soft-Constrained Optimization of Latent Space in Variational Autoencoders

• URL: http://arxiv.org/abs/2607.23751v1
• Date: Sun, 26 Jul 2026 16:52:35 GMT
• Title: Soft-Constrained Optimization of Latent Space in Variational Autoencoders
• Abstract summary: We formulate VAE training as a soft-constrained optimization problem that addresses both.<n>We find that low-entropy discrete factors tend to merge into a single latent variable, whereas high-entropy continuous factors are distributed across several.<n>We also find that low-entropy discrete factors tend to merge into a single latent variable, whereas high-entropy continuous factors are distributed across several.
• Score: 15.27508597551075
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The usefulness of a variational autoencoder (VAE) depends on two properties of its latent space that are hard to obtain together: high encoding capacity in the individual latent variables, and a low-dimensional, disentangled organization of those variables. Weakening the Kullback-Leibler regularization raises capacity but degrades disentanglement, while strengthening it prunes latent variables away entirely. We formulate VAE training as a soft-constrained optimization problem that addresses both. First, we impose an entropy-based constraint (EC) on individual latent variables, showing that the entropy of a latent code upper-bounds the mutual information it carries about the generative factors of the data. Second, we propose a weight-filter method that exploits the slack of the soft constraint to prune low-entropy dimensions during downstream training. On dSprites, the EC raises the aggregate latent-variable activation score by 43-62% over a vanilla VAE, attains the highest FactorVAE score among the \b{eta} \b{eta}-VAE variants (0.891 vs 0.847), and lowers reconstruction error by up to 38%. On MNIST, the weight filter reduces the latent dimensionality supplied to a downstream classifier from ten to two while holding accuracy above 90%, converging in 37% fewer epochs than the same procedure without the EC. We also find that low-entropy discrete factors tend to merge into a single latent variable, whereas high-entropy continuous factors are distributed across several.

Related papers

• A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM [40.07171947823021]
  Chain-of-Thought (CoT) improves the reasoning ability of Large Language Models (LLMs) but incurs substantial computation and context costs.<n>We present A*-Thought-V2, a geometric dynamics of LLM guided framework that models CoT as a hidden-state trajectory and replaces hard deletion with an explicit-implicit interleaved latent architecture.
  arXiv  Detail & Related papers  (2026-09-07T17:56:20Z)
• A Multi-stage Constrained Optimization Framework for Data-driven Problems [15.27508597551075]
  Variational autoencoders (VAEs) transform high-dimensional, often noisy data into a compact latent representation.<n>Three challenges persist in VAE-based constrained optimization.<n>We propose a Multi-stage Constrained Optimization Framework (MCOF)
  arXiv  Detail & Related papers  (2026-07-26T06:11:06Z)
• AOT-POT: Adaptive Operator Transformation for Large-Scale PDE Pre-training [50.28798360093496]
  We propose to transform complex and diverse solution operators into simpler, better-aligned forms that are easier to model jointly.<n>We instantiate this idea as AOT-POT (adaptive operator-transformation for pre-training operator transformer), which expands hidden representations into multiple parallel streams.<n>AOT-POT achieves state-of-the-art performance on 12 PDE benchmarks with only 3% additional parameters, reducing relative L2 error by up to 77.6% (40.9% on average)
  arXiv  Detail & Related papers  (2026-05-15T09:50:34Z)
• Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting [78.96382179408054]
  We introduce bfLeGS, a framework that reformulates density control as a parameterized policy network optimized via Reinforcement Learning (RL)<n>To maintain computational tractability, we derive a closed-form solution that reduces the complexity of reward calculation from $O(N2)$ to $O(N)$.<n>Experiments on the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets demonstrate that bfLeGS significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-01T05:04:49Z)
• LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamics [6.953554594702111]
  We propose Latent Ensemble Data Assimilation (LEVDA), an ensemble-space variational smoother.<n>It assimilates states and unknown parameters without the need for adjoint code or auxiliary observation-to-latent encoders.<n>It substantially improved assimilation accuracy and computational efficiency compared to full-state 4DEnVar.
  arXiv  Detail & Related papers  (2026-02-23T00:54:59Z)
• Towards Unified and Lossless Latent Space for 3D Molecular Latent Diffusion Modeling [90.23688195918432]
  3D molecule generation is crucial for drug discovery and material science.<n>Existing approaches typically maintain separate latent spaces for invariant and equivariant modalities.<n>We propose textbfUAE-3D, a multi-modal VAE that compresses 3D molecules into latent sequences from a unified latent space.
  arXiv  Detail & Related papers  (2025-03-19T08:56:13Z)
• Variantional autoencoder with decremental information bottleneck for disentanglement [16.93743613675349]
  We present a novel framework for disentangled representation learning, DeVAE, which utilizes hierarchical latent spaces with decreasing information bottlenecks. The key innovation of our approach lies in connecting the hierarchical latent spaces through disentanglement-invariant transformations. We demonstrate the effectiveness of DeVAE in achieving a balance between disentanglement and reconstruction through a series of experiments and ablation studies on dSprites and Shapes3D datasets.
  arXiv  Detail & Related papers  (2023-03-22T23:38:10Z)
• Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational AutoEncoder for Diverse Text Generation [85.5379146125199]
  Variational Auto-Encoder (VAE) has been widely adopted in text generation. We propose TRACE, a Transformer-based recurrent VAE structure.
  arXiv  Detail & Related papers  (2022-10-22T10:25:35Z)
• Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation [85.5379146125199]
  We propose a novel variational Transformer framework to overcome the KL vanishing problem. We show that our method can be regarded as entangling latent variables to avoid posterior information decrease through layers.
  arXiv  Detail & Related papers  (2022-07-13T11:27:46Z)
• Double Control Variates for Gradient Estimation in Discrete Latent Variable Models [32.33171301923846]
  We introduce a variance reduction technique for score function estimators. We show that our estimator can have lower variance compared to other state-of-the-art estimators.
  arXiv  Detail & Related papers  (2021-11-09T18:02:42Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• VAE-KRnet and its applications to variational Bayes [4.9545850065593875]
  We have proposed a generative model, called VAE-KRnet, for density estimation or approximation. VAE is used a dimension reduction technique to capture the latent space, and KRnet is used to model the distribution of the latent variable. VAE-KRnet can be used as a density model to approximate either data distribution or an arbitrary probability density function.
  arXiv  Detail & Related papers  (2020-06-29T23:14:36Z)
• Preventing Posterior Collapse with Levenshtein Variational Autoencoder [61.30283661804425]
  We propose to replace the evidence lower bound (ELBO) with a new objective which is simple to optimize and prevents posterior collapse. We show that Levenstein VAE produces more informative latent representations than alternative approaches to preventing posterior collapse.
  arXiv  Detail & Related papers  (2020-04-30T13:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.