ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on
Nonlinear ICA
- URL: http://arxiv.org/abs/2002.11537v4
- Date: Mon, 26 Oct 2020 17:49:11 GMT
- Title: ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on
Nonlinear ICA
- Authors: Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, Aapo
Hyv\"arinen
- Abstract summary: We consider the identifiability theory of probabilistic models.
We show that our model can be used for the estimation of the components in the framework of Independently Modulated Component Analysis.
- Score: 11.919315372249802
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We consider the identifiability theory of probabilistic models and establish
sufficient conditions under which the representations learned by a very broad
family of conditional energy-based models are unique in function space, up to a
simple transformation. In our model family, the energy function is the
dot-product between two feature extractors, one for the dependent variable, and
one for the conditioning variable. We show that under mild conditions, the
features are unique up to scaling and permutation. Our results extend recent
developments in nonlinear ICA, and in fact, they lead to an important
generalization of ICA models. In particular, we show that our model can be used
for the estimation of the components in the framework of Independently
Modulated Component Analysis (IMCA), a new generalization of nonlinear ICA that
relaxes the independence assumption. A thorough empirical study shows that
representations learned by our model from real-world image datasets are
identifiable, and improve performance in transfer learning and semi-supervised
learning tasks.
Related papers
- Identifiable Representation and Model Learning for Latent Dynamic Systems [0.0]
We study the problem of identifiable representation and model learning for latent dynamic systems.
We prove that, for linear or affine nonlinear latent dynamic systems, it is possible to identify the representations up to scaling and determine the models up to some simple transformations.
arXiv Detail & Related papers (2024-10-23T13:55:42Z) - Latent Space Energy-based Neural ODEs [73.01344439786524]
This paper introduces a novel family of deep dynamical models designed to represent continuous-time sequence data.
We train the model using maximum likelihood estimation with Markov chain Monte Carlo.
Experiments on oscillating systems, videos and real-world state sequences (MuJoCo) illustrate that ODEs with the learnable energy-based prior outperform existing counterparts.
arXiv Detail & Related papers (2024-09-05T18:14:22Z) - Shape Arithmetic Expressions: Advancing Scientific Discovery Beyond Closed-Form Equations [56.78271181959529]
Generalized Additive Models (GAMs) can capture non-linear relationships between variables and targets, but they cannot capture intricate feature interactions.
We propose Shape Expressions Arithmetic ( SHAREs) that fuses GAM's flexible shape functions with the complex feature interactions found in mathematical expressions.
We also design a set of rules for constructing SHAREs that guarantee transparency of the found expressions beyond the standard constraints.
arXiv Detail & Related papers (2024-04-15T13:44:01Z) - CoCoGen: Physically-Consistent and Conditioned Score-based Generative Models for Forward and Inverse Problems [1.0923877073891446]
This work extends the reach of generative models into physical problem domains.
We present an efficient approach to promote consistency with the underlying PDE.
We showcase the potential and versatility of score-based generative models in various physics tasks.
arXiv Detail & Related papers (2023-12-16T19:56:10Z) - Discovering Interpretable Physical Models using Symbolic Regression and
Discrete Exterior Calculus [55.2480439325792]
We propose a framework that combines Symbolic Regression (SR) and Discrete Exterior Calculus (DEC) for the automated discovery of physical models.
DEC provides building blocks for the discrete analogue of field theories, which are beyond the state-of-the-art applications of SR to physical problems.
We prove the effectiveness of our methodology by re-discovering three models of Continuum Physics from synthetic experimental data.
arXiv Detail & Related papers (2023-10-10T13:23:05Z) - Flow Factorized Representation Learning [109.51947536586677]
We introduce a generative model which specifies a distinct set of latent probability paths that define different input transformations.
We show that our model achieves higher likelihoods on standard representation learning benchmarks while simultaneously being closer to approximately equivariant models.
arXiv Detail & Related papers (2023-09-22T20:15:37Z) - Is Model Ensemble Necessary? Model-based RL via a Single Model with
Lipschitz Regularized Value Function [23.255250192599327]
Probabilistic dynamics model ensemble is widely used in existing model-based reinforcement learning methods.
We find that, for a value function, the stronger the Lipschitz condition is, the smaller the gap between the true dynamics-induced Bellman operators is.
arXiv Detail & Related papers (2023-02-02T17:27:16Z) - Latent Variable Representation for Reinforcement Learning [131.03944557979725]
It remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of model-based reinforcement learning.
We provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle.
In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models.
arXiv Detail & Related papers (2022-12-17T00:26:31Z) - Indeterminacy in Latent Variable Models: Characterization and Strong
Identifiability [3.959606869996233]
We construct a theoretical framework for analyzing the indeterminacies of latent variable models.
We then investigate how we might specify strongly identifiable latent variable models.
arXiv Detail & Related papers (2022-06-02T00:01:27Z) - I Don't Need $\mathbf{u}$: Identifiable Non-Linear ICA Without Side
Information [13.936583337756883]
We introduce a new approach for identifiable non-linear ICA models.
In particular, we focus on generative models which perform clustering in their latent space.
arXiv Detail & Related papers (2021-06-09T17:22:08Z) - Learning Discrete Energy-based Models via Auxiliary-variable Local
Exploration [130.89746032163106]
We propose ALOE, a new algorithm for learning conditional and unconditional EBMs for discrete structured data.
We show that the energy function and sampler can be trained efficiently via a new variational form of power iteration.
We present an energy model guided fuzzer for software testing that achieves comparable performance to well engineered fuzzing engines like libfuzzer.
arXiv Detail & Related papers (2020-11-10T19:31:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.