Learning Bijective Feature Maps for Linear ICA

• URL: http://arxiv.org/abs/2002.07766v5
• Date: Fri, 29 Jan 2021 18:08:28 GMT
• Title: Learning Bijective Feature Maps for Linear ICA
• Authors: Alexander Camuto, Matthew Willetts, Brooks Paige, Chris Holmes, Stephen Roberts
• Abstract summary: We show that existing probabilistic deep generative models (DGMs) which are tailor-made for image data, underperform on non-linear ICA tasks. To address this, we propose a DGM which combines bijective feature maps with a linear ICA model to learn interpretable latent structures for high-dimensional data. We create models that converge quickly, are easy to train, and achieve better unsupervised latent factor discovery than flow-based models, linear ICA, and Variational Autoencoders on images.
• Score: 73.85904548374575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Separating high-dimensional data like images into independent latent factors, i.e independent component analysis (ICA), remains an open research problem. As we show, existing probabilistic deep generative models (DGMs), which are tailor-made for image data, underperform on non-linear ICA tasks. To address this, we propose a DGM which combines bijective feature maps with a linear ICA model to learn interpretable latent structures for high-dimensional data. Given the complexities of jointly training such a hybrid model, we introduce novel theory that constrains linear ICA to lie close to the manifold of orthogonal rectangular matrices, the Stiefel manifold. By doing so we create models that converge quickly, are easy to train, and achieve better unsupervised latent factor discovery than flow-based models, linear ICA, and Variational Autoencoders on images.

Related papers

• Unbiased Learning of Deep Generative Models with Structured Discrete Representations [7.9057320008285945]
  We propose novel algorithms for learning structured variational autoencoders (SVAEs) We are the first to demonstrate the SVAE's ability to handle multimodal uncertainty when data is missing by incorporating discrete latent variables. Our memory-efficient implicit differentiation scheme makes the SVAE tractable to learn via gradient descent, while demonstrating robustness to incomplete optimization.
  arXiv  Detail & Related papers  (2023-06-14T03:59:21Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• Latent Graph Inference using Product Manifolds [0.0]
  We generalize the discrete Differentiable Graph Module (dDGM) for latent graph learning. Our novel approach is tested on a wide range of datasets, and outperforms the original dDGM model.
  arXiv  Detail & Related papers  (2022-11-26T22:13:06Z)
• Towards a mathematical understanding of learning from few examples with nonlinear feature maps [68.8204255655161]
  We consider the problem of data classification where the training set consists of just a few data points. We reveal key relationships between the geometry of an AI model's feature space, the structure of the underlying data distributions, and the model's generalisation capabilities.
  arXiv  Detail & Related papers  (2022-11-07T14:52:58Z)
• NOMAD: Nonlinear Manifold Decoders for Operator Learning [17.812064311297117]
  Supervised learning in function spaces is an emerging area of machine learning research. We show NOMAD, a novel operator learning framework with a nonlinear decoder map capable of learning finite dimensional representations of nonlinear submanifolds in function spaces.
  arXiv  Detail & Related papers  (2022-06-07T19:52:44Z)
• Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensions [25.635225717360466]
  We show that for a large class of kernels, including the neural kernel of fully connected networks, kernel methods can only perform as well as linear models in a certain high-dimensional regime. More complex models for the data other than independent features are needed for high-dimensional analysis.
  arXiv  Detail & Related papers  (2022-01-20T09:35:46Z)
• Graph Embedding via High Dimensional Model Representation for Hyperspectral Images [9.228929858529678]
  Learning the manifold structure of remote sensing images is of paramount relevance for modeling and understanding processes. Manor learning methods have shown excellent performance to deal with hyperspectral image (HSI) analysis. A common assumption to deal with the problem is that the transformation between the high-dimensional input space and the (typically low) latent space is linear. The proposed method is compared to manifold learning methods along with its linear counterparts and achieves promising performance in terms of classification accuracy of a representative set of hyperspectral images.
  arXiv  Detail & Related papers  (2021-11-29T16:42:15Z)
• Model-agnostic multi-objective approach for the evolutionary discovery of mathematical models [55.41644538483948]
  In modern data science, it is more interesting to understand the properties of the model, which parts could be replaced to obtain better results. We use multi-objective evolutionary optimization for composite data-driven model learning to obtain the algorithm's desired properties.
  arXiv  Detail & Related papers  (2021-07-07T11:17:09Z)
• Mix Dimension in Poincar\'{e} Geometry for 3D Skeleton-based Action Recognition [57.98278794950759]
  Graph Convolutional Networks (GCNs) have already demonstrated their powerful ability to model the irregular data. We present a novel spatial-temporal GCN architecture which is defined via the Poincar'e geometry. We evaluate our method on two current largest scale 3D datasets.
  arXiv  Detail & Related papers  (2020-07-30T18:23:18Z)
• Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
  Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization. We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise. We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
  arXiv  Detail & Related papers  (2020-06-11T17:29:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.