Posterior Collapse as Automatic Spectral Pruning

• URL: http://arxiv.org/abs/2605.22691v1
• Date: Thu, 21 May 2026 16:36:10 GMT
• Title: Posterior Collapse as Automatic Spectral Pruning
• Abstract summary: We show that posterior collapse in $$-VAEs implements automatic spectral pruning.<n>A latent mode collapses if its contribution to reconstruction is below the cutoff set by $$.<n>We derive this as a consequence of the loss via a Landau stability analysis.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We show that posterior collapse in $β$-VAEs implements automatic spectral pruning. A latent mode collapses if its contribution to reconstruction is below the cutoff set by $β$. Equilibrium solutions with different $β$ thus reveal a cascade of collapses as latent modes decouple from least to most useful. We derive this as a consequence of the loss via a Landau stability analysis. We define a latent-rescaling-invariant order parameter that ranks active latent modes and whose collapse thresholds identify which effective variables to inspect first. In the linear Gaussian case, the collapse spectrum, utility spectrum, and normalized PCA spectrum coincide, and each collapse follows a mean-field law. We test these predictions on the WorldClim dataset.

Related papers

• $β$-VAEs as Effective Theories: Tolerance-Dependent Dimension [0.0]
  In a $$-VAE, increasing the regularization strength acts as a spectral cutoff by collapsing low-utility latent coordinates.<n>We find that nonlinear interactions shift and broaden collapse onsets, so thresholds no longer coincide exactly with utilities.<n>The resulting effective-dimension curves reveal a head--tail tradeoff: increasing depth concentrates utility into the first few coordinates but worsens tail fidelity.
  arXiv  Detail & Related papers  (2026-08-11T07:34:18Z)
• Optimization Geometrodynamics: Variational Reduction and Interaction Curvature [0.0]
  We develop geometrodynamics as a variational theory of this hidden determinant pushforward.<n>For affine pre-reductions, the induced interaction curvature is the negative-semidefinite operator $-G*H-1G$.<n>We prove a global analytic bundle with closed totally geodesic fibers and a unique analytic nearest-controller section.
  arXiv  Detail & Related papers  (2026-07-07T18:42:00Z)
• $\mathbfλ$-VAE: Variance Equalization for Posterior Collapse [0.0]
  Variational Autoencoders (VAEs) frequently suffer from posterior collapse, a failure mode in which the approximate posterior converges to the prior.<n>We identify and formalize two logically independent but coupled causes.
  arXiv  Detail & Related papers  (2026-07-06T18:07:46Z)
• Is Spurious Correlation Removal Always Learnable? [56.28155520961125]
  Invariant learning can fail even when the structure is statistically identifiable.<n>We show a conditional computational barrier: under a black-box samplable supervised sparse recovery primitive, there exist emphamplable multi-dimensional environments.<n>Synthetic and real datasets illustrate the predicted gaps and transitions and motivate simple diversity diagnostics.
  arXiv  Detail & Related papers  (2026-06-11T05:49:43Z)
• How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis [53.063298916923976]
  We study the feedback in a Gaussian single-index model with $r*(x) = *(langle *, xrangle)$ and $x sim N(0, I_d)$.<n>We analyze a two-stage neural reward model that first learns the hidden direction $*$ from reward-weighted samples and then fits the readout layer by weighted ridge regression.
  arXiv  Detail & Related papers  (2026-05-23T22:00:38Z)
• How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models [6.758792133077052]
  We derive exact characterizations of the limiting eigenvalue distribution, outlier eigenvalues, and eigenvector alignment with the hidden signal.<n>We show that parameter-free causal self-attention with $/d$ score scaling yields deterministic harmonic weights that improve signal recovery over mean pooling.
  arXiv  Detail & Related papers  (2026-05-07T18:28:01Z)
• Information-Geometric Decomposition of Generalization Error in Unsupervised Learning [0.0]
  We decompose the Kullback--Leibler generalization error (GE) of unsupervised learning into three non-negative components: model error, data bias, and variance.<n>The decomposition is exact for any e-flat model class.
  arXiv  Detail & Related papers  (2026-04-14T06:23:18Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Mutual Information Collapse Explains Disentanglement Failure in $β$-VAEs [4.155522769716163]
  $$-VAE is a framework for unsupervised disentanglement.<n> benchmarks such as MIG and SAP typically peak at intermediate $$ and collapse as regularization increases.<n>We introduce the $$-VAE, which decouples regularization pressure from informational collapse.
  arXiv  Detail & Related papers  (2026-02-09T23:38:11Z)
• Stabilizing Fixed-Point Iteration for Markov Chain Poisson Equations [49.702772230127465]
  We study finite-state Markov chains with $n$ states and transition matrix $P$.<n>We show that all non-decaying modes are captured by a real peripheral invariant subspace $mathcalK(P)$, and that the induced operator on the quotient space $mathbbRn/mathcalK(P) is strictly contractive, yielding a unique quotient solution.
  arXiv  Detail & Related papers  (2026-01-31T02:57:01Z)
• Learning-Based Shrinking Disturbance-Invariant Tubes for State- and Input-Dependent Uncertainty [1.3750624267664155]
  We develop a learning-based framework for constructing shrinking disturbance-invariant tubes under state- and input-dependent uncertainty.<n>We certify safety via a lifted, isotone (order-preserving) fixed-point map.
  arXiv  Detail & Related papers  (2026-01-16T16:47:04Z)
• Optimal policy evaluation using kernel-based temporal difference methods [78.83926562536791]
  We use kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process. We derive a non-asymptotic upper bound on the error with explicit dependence on the eigenvalues of the associated kernel operator. We prove minimax lower bounds over sub-classes of MRPs.
  arXiv  Detail & Related papers  (2021-09-24T14:48:20Z)
• Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance [14.06947898164194]
  Heavy tails emerge in gradient descent (SGD) in various scenarios. We provide convergence guarantees for SGD under a state-dependent and heavy-tailed noise with a potentially infinite variance. Our results indicate that even under heavy-tailed noise with infinite variance, SGD can converge to the global optimum.
  arXiv  Detail & Related papers  (2021-02-20T13:45:11Z)
• Preventing Posterior Collapse with Levenshtein Variational Autoencoder [61.30283661804425]
  We propose to replace the evidence lower bound (ELBO) with a new objective which is simple to optimize and prevents posterior collapse. We show that Levenstein VAE produces more informative latent representations than alternative approaches to preventing posterior collapse.
  arXiv  Detail & Related papers  (2020-04-30T13:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.