Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning

• URL: http://arxiv.org/abs/2303.16535v2
• Date: Tue, 5 Sep 2023 08:45:28 GMT
• Title: Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning
• Authors: Aapo Hyvarinen, Ilyes Khemakhem, Hiroshi Morioka
• Abstract summary: A central problem in unsupervised deep learning is how to find useful representations of high-dimensional data, sometimes called "disentanglement" This paper reviews the state-of-the-art of nonlinear ICA theory and algorithms.
• Score: 2.2329417756084093
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: A central problem in unsupervised deep learning is how to find useful representations of high-dimensional data, sometimes called "disentanglement". Most approaches are heuristic and lack a proper theoretical foundation. In linear representation learning, independent component analysis (ICA) has been successful in many applications areas, and it is principled, i.e., based on a well-defined probabilistic model. However, extension of ICA to the nonlinear case has been problematic due to the lack of identifiability, i.e., uniqueness of the representation. Recently, nonlinear extensions that utilize temporal structure or some auxiliary information have been proposed. Such models are in fact identifiable, and consequently, an increasing number of algorithms have been developed. In particular, some self-supervised algorithms can be shown to estimate nonlinear ICA, even though they have initially been proposed from heuristic perspectives. This paper reviews the state-of-the-art of nonlinear ICA theory and algorithms.

Related papers

• Limits and Powers of Koopman Learning [0.0]
  Dynamical systems provide a comprehensive way to study complex and changing behaviors across various sciences. Koopman operators have emerged as a dominant approach because they allow the study of nonlinear dynamics using linear techniques. This paper addresses a fundamental open question: textitWhen can we robustly learn the spectral properties of Koopman operators from trajectory data of dynamical systems, and when can we not?
  arXiv  Detail & Related papers  (2024-07-08T18:24:48Z)
• Identifiable Feature Learning for Spatial Data with Nonlinear ICA [18.480534062833673]
  We introduce a new nonlinear ICA framework that employs latent components which apply naturally to data with higher-dimensional dependency structures. In particular, we develop a new learning and algorithm that extends variational methods to handle the combination of a deep neural network mixing function with the TP prior inducing computational efficacy.
  arXiv  Detail & Related papers  (2023-11-28T15:00:11Z)
• Learning Linear Causal Representations from Interventions under General Nonlinear Mixing [52.66151568785088]
  We prove strong identifiability results given unknown single-node interventions without access to the intervention targets. This is the first instance of causal identifiability from non-paired interventions for deep neural network embeddings.
  arXiv  Detail & Related papers  (2023-06-04T02:32:12Z)
• On the Identifiability of Nonlinear ICA: Sparsity and Beyond [20.644375143901488]
  How to make the nonlinear ICA model identifiable up to certain trivial indeterminacies is a long-standing problem in unsupervised learning. Recent breakthroughs reformulate the standard independence assumption of sources as conditional independence given some auxiliary variables. We show that under specific instantiations of such constraints, the independent latent sources can be identified from their nonlinear mixtures up to a permutation.
  arXiv  Detail & Related papers  (2022-06-15T18:24:22Z)
• Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning [37.27098255569438]
  We study the role of nonlinearity in the training dynamics of contrastive learning (CL) on one and two-layer nonlinear networks. We show that the presence of nonlinearity leads to many local optima even in 1-layer setting. For 2-layer setting, we also discover emphglobal modulation: those local patterns discriminative from the perspective of global-level patterns are prioritized to learn.
  arXiv  Detail & Related papers  (2022-06-02T23:52:35Z)
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms [71.62575565990502]
  We prove that the generalization error of an optimization algorithm can be bounded on the complexity' of the fractal structure that underlies its generalization measure. We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden/layered neural networks) and algorithms.
  arXiv  Detail & Related papers  (2021-06-09T08:05:36Z)
• Hessian Eigenspectra of More Realistic Nonlinear Models [73.31363313577941]
  We make a emphprecise characterization of the Hessian eigenspectra for a broad family of nonlinear models. Our analysis takes a step forward to identify the origin of many striking features observed in more complex machine learning models.
  arXiv  Detail & Related papers  (2021-03-02T06:59:52Z)
• Learning Fast Approximations of Sparse Nonlinear Regression [50.00693981886832]
  In this work, we bridge the gap by introducing the Threshold Learned Iterative Shrinkage Algorithming (NLISTA) Experiments on synthetic data corroborate our theoretical results and show our method outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-10-26T11:31:08Z)
• Nonlinear ISA with Auxiliary Variables for Learning Speech Representations [51.9516685516144]
  We introduce a theoretical framework for nonlinear Independent Subspace Analysis (ISA) in the presence of auxiliary variables. We propose an algorithm that learns unsupervised speech representations whose subspaces are independent.
  arXiv  Detail & Related papers  (2020-07-25T14:53:09Z)
• Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series [0.0]
  We show how to combine nonlinear Independent Component Analysis with a Hidden Markov Model. We prove identifiability of the proposed model for a general mixing nonlinearity, such as a neural network. We achieve a new nonlinear ICA framework which is unsupervised, more efficient, as well as able to model underlying temporal dynamics.
  arXiv  Detail & Related papers  (2020-06-22T10:01:15Z)
• Eigendecomposition-Free Training of Deep Networks for Linear Least-Square Problems [107.3868459697569]
  We introduce an eigendecomposition-free approach to training a deep network. We show that our approach is much more robust than explicit differentiation of the eigendecomposition. Our method has better convergence properties and yields state-of-the-art results.
  arXiv  Detail & Related papers  (2020-04-15T04:29:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.