Disentangling Granularity: An Implicit Inductive Bias in Factorized VAEs

• URL: http://arxiv.org/abs/2505.24684v1
• Date: Fri, 30 May 2025 15:08:50 GMT
• Title: Disentangling Granularity: An Implicit Inductive Bias in Factorized VAEs
• Authors: Zihao Chen, Yu Xiang, Wenyong Wang,
• Abstract summary: We study the implicit inductive bias that drive disentanglement in variational autoencoders (VAEs) with factorization priors.<n>We show that disentangling granularity as an implicit inductive bias in factorized VAEs influence both disentanglement performance and the inference of the Evidence Lower Bound (ELBO)<n>Our findings unveil that disentangling granularity as an implicit inductive bias in factorized VAEs influence both disentanglement performance and the inference of the ELBO, offering fresh insights into the interpretability and inherent biases of VAEs.
• Score: 4.987314374901578
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the success in learning semantically meaningful, unsupervised disentangled representations, variational autoencoders (VAEs) and their variants face a fundamental theoretical challenge: substantial evidence indicates that unsupervised disentanglement is unattainable without implicit inductive bias, yet such bias remains elusive. In this work, we focus on exploring the implicit inductive bias that drive disentanglement in VAEs with factorization priors. By analyzing the total correlation in \b{eta}-TCVAE, we uncover a crucial implicit inductive bias called disentangling granularity, which leads to the discovery of an interesting "V"-shaped optimal Evidence Lower Bound (ELBO) trajectory within the parameter space. This finding is validated through over 100K experiments using factorized VAEs and our newly proposed model, \b{eta}-STCVAE. Notably, experimental results reveal that conventional factorized VAEs, constrained by fixed disentangling granularity, inherently tend to disentangle low-complexity feature. Whereas, appropriately tuning disentangling granularity, as enabled by \b{eta}-STCVAE, broadens the range of disentangled representations, allowing for the disentanglement of high-complexity features. Our findings unveil that disentangling granularity as an implicit inductive bias in factorized VAEs influence both disentanglement performance and the inference of the ELBO, offering fresh insights into the interpretability and inherent biases of VAEs.

Related papers

• A Non-negative VAE:the Generalized Gamma Belief Network [49.970917207211556]
  The gamma belief network (GBN) has demonstrated its potential for uncovering multi-layer interpretable latent representations in text data. We introduce the generalized gamma belief network (Generalized GBN) in this paper, which extends the original linear generative model to a more expressive non-linear generative model. We also propose an upward-downward Weibull inference network to approximate the posterior distribution of the latent variables.
  arXiv  Detail & Related papers  (2024-08-06T18:18:37Z)
• Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Bias [8.668428992331808]
  We develop an Sobolev norm learning curve for kernel ridge(less) regression when addressing (elliptical) linear inverse problems.<n>Our results show that the PDE operators in the inverse problem can stabilize the variance and even behave benign overfitting for fixed-dimensional problems.
  arXiv  Detail & Related papers  (2024-06-13T14:54:30Z)
• Closed-Loop Unsupervised Representation Disentanglement with $\eta$-VAE Distillation and Diffusion Probabilistic Feedback [45.68054456449699]
  Representation disentanglement may help AI fundamentally understand the real world and thus benefit both discrimination and generation tasks. We propose a textbfCL-Disentanglement approach dubbed textbfCL-Dis. Experiments demonstrate the superiority of CL-Dis on applications like real image manipulation and visual analysis.
  arXiv  Detail & Related papers  (2024-02-04T05:03:22Z)
• Learning Disentangled Discrete Representations [22.5004558029479]
  We show the relationship between discrete latent spaces and disentangled representations by replacing the standard Gaussian variational autoencoder with a tailored categorical variational autoencoder. We provide both analytical and empirical findings that demonstrate the advantages of discrete VAEs for learning disentangled representations.
  arXiv  Detail & Related papers  (2023-07-26T12:29:58Z)
• Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Models [5.519653885553456]
  Generalized Additive Models (GAMs) have recently experienced a resurgence in popularity due to their interpretability. We show how concurvity can severly impair the interpretability of GAMs. We propose a remedy: a conceptually simple, yet effective regularizer which penalizes pairwise correlations of the non-linearly transformed feature variables.
  arXiv  Detail & Related papers  (2023-05-19T06:55:49Z)
• Strong inductive biases provably prevent harmless interpolation [8.946655323517092]
  This paper argues that the degree to which is harmless hinges upon the strength of an estimator's inductive bias. Our main theoretical result establishes tight non-asymptotic bounds for high-dimensional kernel regression.
  arXiv  Detail & Related papers  (2023-01-18T15:37:11Z)
• Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers [52.468311268601056]
  This paper analyzes attention through the lens of convex duality. We derive equivalent finite-dimensional convex problems that are interpretable and solvable to global optimality. We show how self-attention networks implicitly cluster the tokens, based on their latent similarity.
  arXiv  Detail & Related papers  (2022-05-17T04:01:15Z)
• Deconfounded Score Method: Scoring DAGs with Dense Unobserved Confounding [101.35070661471124]
  We show that unobserved confounding leaves a characteristic footprint in the observed data distribution that allows for disentangling spurious and causal effects. We propose an adjusted score-based causal discovery algorithm that may be implemented with general-purpose solvers and scales to high-dimensional problems.
  arXiv  Detail & Related papers  (2021-03-28T11:07:59Z)
• Efficient Semi-Implicit Variational Inference [65.07058307271329]
  We propose an efficient and scalable semi-implicit extrapolational (SIVI) Our method maps SIVI's evidence to a rigorous inference of lower gradient values.
  arXiv  Detail & Related papers  (2021-01-15T11:39:09Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.