Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA

• URL: http://arxiv.org/abs/2106.09620v1
• Date: Thu, 17 Jun 2021 15:56:57 GMT
• Title: Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA
• Authors: Hermanni H\"alv\"a, Sylvain Le Corff, Luc Leh\'ericy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, Aapo Hyvarinen
• Abstract summary: We introduce a new general identifiable framework for principled disentanglement referred to as Structured Independent Component Analysis (SNICA) Our contribution is to extend the identifiability theory of deep generative models for a very broad class of structured models. We establish the major result that identifiability for this framework holds even in the presence of noise of unknown distribution.
• Score: 4.340954888479091
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a new general identifiable framework for principled disentanglement referred to as Structured Nonlinear Independent Component Analysis (SNICA). Our contribution is to extend the identifiability theory of deep generative models for a very broad class of structured models. While previous works have shown identifiability for specific classes of time-series models, our theorems extend this to more general temporal structures as well as to models with more complex structures such as spatial dependencies. In particular, we establish the major result that identifiability for this framework holds even in the presence of noise of unknown distribution. The SNICA setting therefore subsumes all the existing nonlinear ICA models for time-series and also allows for new much richer identifiable models. Finally, as an example of our framework's flexibility, we introduce the first nonlinear ICA model for time-series that combines the following very useful properties: it accounts for both nonstationarity and autocorrelation in a fully unsupervised setting; performs dimensionality reduction; models hidden states; and enables principled estimation and inference by variational maximum-likelihood.

Related papers

• Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks [50.29356570858905]
  We introduce the Dynamical Systems Framework (DSF), which allows a principled investigation of all these architectures in a common representation. We provide principled comparisons between softmax attention and other model classes, discussing the theoretical conditions under which softmax attention can be approximated. This shows the DSF's potential to guide the systematic development of future more efficient and scalable foundation models.
  arXiv  Detail & Related papers  (2024-05-24T17:19:57Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• Indeterminacy in Latent Variable Models: Characterization and Strong Identifiability [3.959606869996233]
  We construct a theoretical framework for analyzing the indeterminacies of latent variable models. We then investigate how we might specify strongly identifiable latent variable models.
  arXiv  Detail & Related papers  (2022-06-02T00:01:27Z)
• On generative models as the basis for digital twins [0.0]
  A framework is proposed for generative models as a basis for digital twins or mirrors of structures. The proposal is based on the premise that deterministic models cannot account for the uncertainty present in most structural modelling applications.
  arXiv  Detail & Related papers  (2022-03-08T20:34:56Z)
• Leveraging the structure of dynamical systems for data-driven modeling [111.45324708884813]
  We consider the impact of the training set and its structure on the quality of the long-term prediction. We show how an informed design of the training set, based on invariants of the system and the structure of the underlying attractor, significantly improves the resulting models.
  arXiv  Detail & Related papers  (2021-12-15T20:09:20Z)
• Closed-form Continuous-Depth Models [99.40335716948101]
  Continuous-depth neural models rely on advanced numerical differential equation solvers. We present a new family of models, termed Closed-form Continuous-depth (CfC) networks, that are simple to describe and at least one order of magnitude faster.
  arXiv  Detail & Related papers  (2021-06-25T22:08:51Z)
• I Don't Need $\mathbf{u}$: Identifiable Non-Linear ICA Without Side Information [13.936583337756883]
  We introduce a new approach for identifiable non-linear ICA models. In particular, we focus on generative models which perform clustering in their latent space.
  arXiv  Detail & Related papers  (2021-06-09T17:22:08Z)
• Hessian Eigenspectra of More Realistic Nonlinear Models [73.31363313577941]
  We make a emphprecise characterization of the Hessian eigenspectra for a broad family of nonlinear models. Our analysis takes a step forward to identify the origin of many striking features observed in more complex machine learning models.
  arXiv  Detail & Related papers  (2021-03-02T06:59:52Z)
• Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder [59.69303945834122]
  We present a Smoothness-Inducing Sequential Variational Auto-Encoder (SISVAE) model for robust estimation and anomaly detection of time series. Our model parameterizes mean and variance for each time-stamp with flexible neural networks. We show the effectiveness of our model on both synthetic datasets and public real-world benchmarks.
  arXiv  Detail & Related papers  (2021-02-02T06:15:15Z)
• Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series [0.0]
  We show how to combine nonlinear Independent Component Analysis with a Hidden Markov Model. We prove identifiability of the proposed model for a general mixing nonlinearity, such as a neural network. We achieve a new nonlinear ICA framework which is unsupervised, more efficient, as well as able to model underlying temporal dynamics.
  arXiv  Detail & Related papers  (2020-06-22T10:01:15Z)
• ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA [11.919315372249802]
  We consider the identifiability theory of probabilistic models. We show that our model can be used for the estimation of the components in the framework of Independently Modulated Component Analysis.
  arXiv  Detail & Related papers  (2020-02-26T14:43:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.