Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning

• URL: http://arxiv.org/abs/2503.00639v1
• Date: Sat, 01 Mar 2025 22:21:37 GMT
• Title: Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning
• Authors: Zijian Li, Shunxing Fan, Yujia Zheng, Ignavier Ng, Shaoan Xie, Guangyi Chen, Xinshuai Dong, Ruichu Cai, Kun Zhang,
• Abstract summary: Disentangled representation learning aims to uncover latent variables underlying the observed data.<n>Some approaches rely on sufficient changes on the distribution of latent variables indicated by auxiliary variables such as domain indices.<n>We propose an identifiability theory with less restrictive constraints regarding distribution changes and the sparse mixing procedure.
• Score: 32.482584125236016
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Disentangled representation learning aims to uncover latent variables underlying the observed data, and generally speaking, rather strong assumptions are needed to ensure identifiability. Some approaches rely on sufficient changes on the distribution of latent variables indicated by auxiliary variables such as domain indices, but acquiring enough domains is often challenging. Alternative approaches exploit structural sparsity assumptions on the mixing procedure, but such constraints are usually (partially) violated in practice. Interestingly, we find that these two seemingly unrelated assumptions can actually complement each other to achieve identifiability. Specifically, when conditioned on auxiliary variables, the sparse mixing procedure assumption provides structural constraints on the mapping from estimated to true latent variables and hence compensates for potentially insufficient distribution changes. Building on this insight, we propose an identifiability theory with less restrictive constraints regarding distribution changes and the sparse mixing procedure, enhancing applicability to real-world scenarios. Additionally, we develop an estimation framework incorporating a domain encoding network and a sparse mixing constraint and provide two implementations based on variational autoencoders and generative adversarial networks, respectively. Experiment results on synthetic and real-world datasets support our theoretical results.

Related papers

• Partial Transportability for Domain Generalization [56.37032680901525]
  Building on the theory of partial identification and transportability, this paper introduces new results for bounding the value of a functional of the target distribution. Our contribution is to provide the first general estimation technique for transportability problems. We propose a gradient-based optimization scheme for making scalable inferences in practice.
  arXiv  Detail & Related papers  (2025-03-30T22:06:37Z)
• Conformal Prediction under Lévy-Prokhorov Distribution Shifts: Robustness to Local and Global Perturbations [41.94295877935867]
  We construct robust conformal prediction intervals that remain valid under distribution shifts. We show that the link between conformal prediction and LP ambiguity sets is a natural one. Building on this analysis, we construct robust conformal prediction intervals that remain valid under distribution shifts.
  arXiv  Detail & Related papers  (2025-02-19T21:18:11Z)
• Towards Understanding Extrapolation: a Causal Lens [53.15488984371969]
  We provide a theoretical understanding of when extrapolation is possible and offer principled methods to achieve it.<n>Under this formulation, we cast the extrapolation problem into a latent-variable identification problem.<n>Our theory reveals the intricate interplay between the underlying manifold's smoothness and the shift properties.
  arXiv  Detail & Related papers  (2025-01-15T21:29:29Z)
• Proxy Methods for Domain Adaptation [78.03254010884783]
  proxy variables allow for adaptation to distribution shift without explicitly recovering or modeling latent variables. We develop a two-stage kernel estimation approach to adapt to complex distribution shifts in both settings.
  arXiv  Detail & Related papers  (2024-03-12T09:32:41Z)
• Identifiable Latent Causal Content for Domain Adaptation under Latent Covariate Shift [82.14087963690561]
  Multi-source domain adaptation (MSDA) addresses the challenge of learning a label prediction function for an unlabeled target domain. We present an intricate causal generative model by introducing latent noises across domains, along with a latent content variable and a latent style variable. The proposed approach showcases exceptional performance and efficacy on both simulated and real-world datasets.
  arXiv  Detail & Related papers  (2022-08-30T11:25:15Z)
• Causal Discovery in Heterogeneous Environments Under the Sparse Mechanism Shift Hypothesis [7.895866278697778]
  Machine learning approaches commonly rely on the assumption of independent and identically distributed (i.i.d.) data. In reality, this assumption is almost always violated due to distribution shifts between environments. We propose the Mechanism Shift Score (MSS), a score-based approach amenable to various empirical estimators.
  arXiv  Detail & Related papers  (2022-06-04T15:39:30Z)
• Structure-preserving GANs [6.438897276587413]
  We introduce structure-preserving GANs as a data-efficient framework for learning distributions. We show that we can reduce the discriminator space to its projection on the invariant discriminator space. We contextualize our framework by building symmetry-preserving GANs for distributions with intrinsic group symmetry.
  arXiv  Detail & Related papers  (2022-02-02T16:40:04Z)
• Which Invariance Should We Transfer? A Causal Minimax Learning Approach [18.71316951734806]
  We present a comprehensive minimax analysis from a causal perspective. We propose an efficient algorithm to search for the subset with minimal worst-case risk. The effectiveness and efficiency of our methods are demonstrated on synthetic data and the diagnosis of Alzheimer's disease.
  arXiv  Detail & Related papers  (2021-07-05T09:07:29Z)
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation [109.73983088432364]
  We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA) We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
  arXiv  Detail & Related papers  (2020-10-09T15:42:35Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.