Towards One Model for Classical Dimensionality Reduction: A Probabilistic Perspective on UMAP and t-SNE

• URL: http://arxiv.org/abs/2405.17412v3
• Date: Thu, 06 Mar 2025 16:22:22 GMT
• Title: Towards One Model for Classical Dimensionality Reduction: A Probabilistic Perspective on UMAP and t-SNE
• Authors: Aditya Ravuri, Neil D. Lawrence,
• Abstract summary: This paper describes the graph Laplacian (an estimate of the data precision matrix) using a Wishart distribution, with a mean given by a non-linear covariance function evaluated on the latents.<n>We introduce tools with which similar dimensionality reduction methods can be studied.
• Score: 8.121681696358717
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper shows that dimensionality reduction methods such as UMAP and t-SNE, can be approximately recast as MAP inference methods corresponding to a model introduced in ProbDR, that describes the graph Laplacian (an estimate of the data precision matrix) using a Wishart distribution, with a mean given by a non-linear covariance function evaluated on the latents. This interpretation offers deeper theoretical and semantic insights into such algorithms, by showing that variances corresponding to these covariances are low (potentially misspecified), and forging a connection to Gaussian process latent variable models by showing that well-known kernels can be used to describe covariances implied by graph Laplacians. We also introduce tools with which similar dimensionality reduction methods can be studied.

Related papers

• Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models [0.0]
  We show how diffusion-based models can be repurposed for performing principled, identifiable Bayesian inference. We show how such maps can be learned via standard DBM training using a novel noise schedule. The result is a class of highly expressive generative models, uniquely defined on a low-dimensional latent space.
  arXiv  Detail & Related papers  (2024-07-11T19:58:19Z)
• Improving Probabilistic Diffusion Models With Optimal Diagonal Covariance Matching [27.2761325416843]
  We leverage the recently proposed covariance moment matching technique and introduce a novel method for learning the diagonal covariance. We demonstrate how our method can substantially enhance the sampling efficiency, recall rate and likelihood of commonly used diffusion models.
  arXiv  Detail & Related papers  (2024-06-16T05:47:12Z)
• Diffusion models for probabilistic programming [56.47577824219207]
  Diffusion Model Variational Inference (DMVI) is a novel method for automated approximate inference in probabilistic programming languages (PPLs) DMVI is easy to implement, allows hassle-free inference in PPLs without the drawbacks of, e.g., variational inference using normalizing flows, and does not make any constraints on the underlying neural network model.
  arXiv  Detail & Related papers  (2023-11-01T12:17:05Z)
• Generalizing Backpropagation for Gradient-Based Interpretability [103.2998254573497]
  We show that the gradient of a model is a special case of a more general formulation using semirings. This observation allows us to generalize the backpropagation algorithm to efficiently compute other interpretable statistics.
  arXiv  Detail & Related papers  (2023-07-06T15:19:53Z)
• A Geometric Perspective on Diffusion Models [57.27857591493788]
  We inspect the ODE-based sampling of a popular variance-exploding SDE. We establish a theoretical relationship between the optimal ODE-based sampling and the classic mean-shift (mode-seeking) algorithm.
  arXiv  Detail & Related papers  (2023-05-31T15:33:16Z)
• Learning Graphical Factor Models with Riemannian Optimization [70.13748170371889]
  This paper proposes a flexible algorithmic framework for graph learning under low-rank structural constraints. The problem is expressed as penalized maximum likelihood estimation of an elliptical distribution. We leverage geometries of positive definite matrices and positive semi-definite matrices of fixed rank that are well suited to elliptical models.
  arXiv  Detail & Related papers  (2022-10-21T13:19:45Z)
• A Provably Efficient Model-Free Posterior Sampling Method for Episodic Reinforcement Learning [50.910152564914405]
  Existing posterior sampling methods for reinforcement learning are limited by being model-based or lack worst-case theoretical guarantees beyond linear MDPs. This paper proposes a new model-free formulation of posterior sampling that applies to more general episodic reinforcement learning problems with theoretical guarantees.
  arXiv  Detail & Related papers  (2022-08-23T12:21:01Z)
• Planning with Diffusion for Flexible Behavior Synthesis [125.24438991142573]
  We consider what it would look like to fold as much of the trajectory optimization pipeline as possible into the modeling problem. The core of our technical approach lies in a diffusion probabilistic model that plans by iteratively denoising trajectories.
  arXiv  Detail & Related papers  (2022-05-20T07:02:03Z)
• Equivariance Discovery by Learned Parameter-Sharing [153.41877129746223]
  We study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme.
  arXiv  Detail & Related papers  (2022-04-07T17:59:19Z)
• A Probabilistic Graph Coupling View of Dimension Reduction [5.35952718937799]
  We introduce a unifying statistical framework based on the coupling of hidden graphs using cross entropy. We show that existing pairwise similarity DR methods can be retrieved from our framework with particular choices of priors for the graphs. Our model is leveraged and extended to address the issue while new links are drawn with Laplacian eigenmaps and PCA.
  arXiv  Detail & Related papers  (2022-01-31T08:21:55Z)
• Improving Metric Dimensionality Reduction with Distributed Topology [68.8204255655161]
  DIPOLE is a dimensionality-reduction post-processing step that corrects an initial embedding by minimizing a loss functional with both a local, metric term and a global, topological term. We observe that DIPOLE outperforms popular methods like UMAP, t-SNE, and Isomap on a number of popular datasets.
  arXiv  Detail & Related papers  (2021-06-14T17:19:44Z)
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories [61.3299263929289]
  Simulation-based inference enables learning the parameters of a model even when its likelihood cannot be computed in practice. One class of methods uses data simulated with different parameters to infer an amortized estimator for the likelihood-to-evidence ratio. We show that this approach can be formulated in terms of mutual information between model parameters and simulated data.
  arXiv  Detail & Related papers  (2021-06-03T12:59:16Z)
• Probabilistic Circuits for Variational Inference in Discrete Graphical Models [101.28528515775842]
  Inference in discrete graphical models with variational methods is difficult. Many sampling-based methods have been proposed for estimating Evidence Lower Bound (ELBO) We propose a new approach that leverages the tractability of probabilistic circuit models, such as Sum Product Networks (SPN) We show that selective-SPNs are suitable as an expressive variational distribution, and prove that when the log-density of the target model is aweighted the corresponding ELBO can be computed analytically.
  arXiv  Detail & Related papers  (2020-10-22T05:04:38Z)
• Estimating Model Uncertainty of Neural Networks in Sparse Information Form [39.553268191681376]
  We present a sparse representation of model uncertainty for Deep Neural Networks (DNNs) The key insight of our work is that the information matrix tends to be sparse in its spectrum. We show that the information form can be scalably applied to represent model uncertainty in DNNs.
  arXiv  Detail & Related papers  (2020-06-20T18:09:59Z)
• Reinforcement Learning as Iterative and Amortised Inference [62.997667081978825]
  We use the control as inference framework to outline a novel classification scheme based on amortised and iterative inference. We show that taking this perspective allows us to identify parts of the algorithmic design space which have been relatively unexplored.
  arXiv  Detail & Related papers  (2020-06-13T16:10:03Z)
• Fitting Laplacian Regularized Stratified Gaussian Models [0.0]
  We consider the problem of jointly estimating multiple related zero-mean Gaussian distributions from data. We propose a distributed method that scales to large problems, and illustrate the efficacy of the method with examples in finance, radar signal processing, and weather forecasting.
  arXiv  Detail & Related papers  (2020-05-04T18:00:59Z)
• Bidirectional Generative Modeling Using Adversarial Gradient Estimation [15.270525239234072]
  We show that different divergences induce similar algorithms in terms of gradient evaluation. This paper gives a general recipe for a class of principled $f$-divergence based generative modeling methods.
  arXiv  Detail & Related papers  (2020-02-21T07:28:56Z)
• Differential Similarity in Higher Dimensional Spaces: Theory and Applications [0.0]
  We develop an algorithm for clustering and coding that combines a geometric model with a probabilistic model in a principled way. We evaluate the solution strategies and the estimation techniques by applying them to two familiar real-world examples.
  arXiv  Detail & Related papers  (2019-02-10T20:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.