Preventing Model Collapse in Gaussian Process Latent Variable Models

• URL: http://arxiv.org/abs/2404.01697v2
• Date: Tue, 18 Jun 2024 08:56:13 GMT
• Title: Preventing Model Collapse in Gaussian Process Latent Variable Models
• Authors: Ying Li, Zhidi Lin, Feng Yin, Michael Minyi Zhang,
• Abstract summary: This paper theoretically examines the impact of projection variance on model collapse through the lens of a linear FourierVM. We tackle model collapse due to inadequate kernel flexibility by integrating the spectral mixture (SM) kernel and a differentiable random feature (RFF) kernel approximation. The proposedVM, named advisedRFLVM, is evaluated across diverse datasets and consistently outperforms various competing models.
• Score: 11.45681373843122
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaussian process latent variable models (GPLVMs) are a versatile family of unsupervised learning models commonly used for dimensionality reduction. However, common challenges in modeling data with GPLVMs include inadequate kernel flexibility and improper selection of the projection noise, leading to a type of model collapse characterized by vague latent representations that do not reflect the underlying data structure. This paper addresses these issues by, first, theoretically examining the impact of projection variance on model collapse through the lens of a linear GPLVM. Second, we tackle model collapse due to inadequate kernel flexibility by integrating the spectral mixture (SM) kernel and a differentiable random Fourier feature (RFF) kernel approximation, which ensures computational scalability and efficiency through off-the-shelf automatic differentiation tools for learning the kernel hyperparameters, projection variance, and latent representations within the variational inference framework. The proposed GPLVM, named advisedRFLVM, is evaluated across diverse datasets and consistently outperforms various salient competing models, including state-of-the-art variational autoencoders (VAEs) and other GPLVM variants, in terms of informative latent representations and missing data imputation.

Related papers

• Scalable Amortized GPLVMs for Single Cell Transcriptomics Data [9.010523724015398]
  Dimensionality reduction is crucial for analyzing large-scale single-cell RNA-seq data. We introduce an improved model, the amortized variational model (BGPLVM) BGPLVM is tailored for single-cell RNA-seq with specialized encoder, kernel, and likelihood designs.
  arXiv  Detail & Related papers  (2024-05-06T21:54:38Z)
• Diffusion models for probabilistic programming [56.47577824219207]
  Diffusion Model Variational Inference (DMVI) is a novel method for automated approximate inference in probabilistic programming languages (PPLs) DMVI is easy to implement, allows hassle-free inference in PPLs without the drawbacks of, e.g., variational inference using normalizing flows, and does not make any constraints on the underlying neural network model.
  arXiv  Detail & Related papers  (2023-11-01T12:17:05Z)
• Bayesian Non-linear Latent Variable Modeling via Random Fourier Features [7.856578780790166]
  We present a method to perform Markov chain Monte Carlo inference for generalized nonlinear latent variable modeling. Inference forVMs is computationally tractable only when the data likelihood is Gaussian. We show that we can generalizeVMs to non-Gaussian observations, such as Poisson, negative binomial, and multinomial distributions.
  arXiv  Detail & Related papers  (2023-06-14T08:42:10Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• Generalised Gaussian Process Latent Variable Models (GPLVM) with Stochastic Variational Inference [9.468270453795409]
  We study the doubly formulation of the BayesianVM model amenable with minibatch training. We show how this framework is compatible with different latent variable formulations and perform experiments to compare a suite of models. We demonstrate how we can train in the presence of massively missing data and obtain high-fidelity reconstructions.
  arXiv  Detail & Related papers  (2022-02-25T21:21:51Z)
• Equivariant vector field network for many-body system modeling [65.22203086172019]
  Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers. We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
  arXiv  Detail & Related papers  (2021-10-26T14:26:25Z)
• Collaborative Nonstationary Multivariate Gaussian Process Model [2.362467745272567]
  We propose a novel model called the collaborative nonstationary Gaussian process model(CNMGP) CNMGP allows us to model data in which outputs do not share a common input set, with a computational complexity independent of the size of the inputs and outputs. We show that our model generally pro-vides better predictive performance than the state-of-the-art, and also provides estimates of time-varying correlations that differ across outputs.
  arXiv  Detail & Related papers  (2021-06-01T18:25:22Z)
• Generalized Matrix Factorization: efficient algorithms for fitting generalized linear latent variable models to large data arrays [62.997667081978825]
  Generalized Linear Latent Variable models (GLLVMs) generalize such factor models to non-Gaussian responses. Current algorithms for estimating model parameters in GLLVMs require intensive computation and do not scale to large datasets. We propose a new approach for fitting GLLVMs to high-dimensional datasets, based on approximating the model using penalized quasi-likelihood.
  arXiv  Detail & Related papers  (2020-10-06T04:28:19Z)
• Accounting for Unobserved Confounding in Domain Generalization [107.0464488046289]
  This paper investigates the problem of learning robust, generalizable prediction models from a combination of datasets. Part of the challenge of learning robust models lies in the influence of unobserved confounders. We demonstrate the empirical performance of our approach on healthcare data from different modalities.
  arXiv  Detail & Related papers  (2020-07-21T08:18:06Z)
• Latent variable modeling with random features [7.856578780790166]
  We develop a family of nonlinear dimension reduction models that are easily to non-Gaussian data likelihoods. Our generalized RFLVMs produce results comparable with other state-of-the-art dimension reduction methods.
  arXiv  Detail & Related papers  (2020-06-19T14:12:05Z)
• Bayesian Sparse Factor Analysis with Kernelized Observations [67.60224656603823]
  Multi-view problems can be faced with latent variable models. High-dimensionality and non-linear issues are traditionally handled by kernel methods. We propose merging both approaches into single model.
  arXiv  Detail & Related papers  (2020-06-01T14:25:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.