Reliable Measures of Spread in High Dimensional Latent Spaces

• URL: http://arxiv.org/abs/2212.08172v2
• Date: Mon, 31 Jul 2023 19:11:04 GMT
• Title: Reliable Measures of Spread in High Dimensional Latent Spaces
• Authors: Anna C. Marbut, Katy McKinney-Bock and Travis J. Wheeler
• Abstract summary: We argue that the commonly used measures of data spread do not provide reliable metrics to compare the use of latent space across models. We propose and examine eight alternative measures of data spread, all but one of which improve over these current metrics. Of our proposed measures, we recommend one principal component-based measure and one entropy-based measure.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Understanding geometric properties of natural language processing models' latent spaces allows the manipulation of these properties for improved performance on downstream tasks. One such property is the amount of data spread in a model's latent space, or how fully the available latent space is being used. In this work, we define data spread and demonstrate that the commonly used measures of data spread, Average Cosine Similarity and a partition function min/max ratio I(V), do not provide reliable metrics to compare the use of latent space across models. We propose and examine eight alternative measures of data spread, all but one of which improve over these current metrics when applied to seven synthetic data distributions. Of our proposed measures, we recommend one principal component-based measure and one entropy-based measure that provide reliable, relative measures of spread and can be used to compare models of different sizes and dimensionalities.

Related papers

• A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models [6.647819824559201]
  We study the large-sample properties of a likelihood-based approach for estimating conditional deep generative models. Our results lead to the convergence rate of a sieve maximum likelihood estimator for estimating the conditional distribution.
  arXiv  Detail & Related papers  (2024-10-02T20:46:21Z)
• (Deep) Generative Geodesics [57.635187092922976]
  We introduce a newian metric to assess the similarity between any two data points. Our metric leads to the conceptual definition of generative distances and generative geodesics. Their approximations are proven to converge to their true values under mild conditions.
  arXiv  Detail & Related papers  (2024-07-15T21:14:02Z)
• Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
  We propose an effective method called Latent Semantic Consensus (LSC) LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses. LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
  arXiv  Detail & Related papers  (2024-03-11T05:35:38Z)
• Metric Space Magnitude for Evaluating the Diversity of Latent Representations [13.272500655475486]
  We develop a family of magnitude-based measures of the intrinsic diversity of latent representations. Our measures are provably stable under perturbations of the data, can be efficiently calculated, and enable a rigorous multi-scale characterisation and comparison of latent representations. We show their utility and superior performance across different domains and tasks, including (i) the automated estimation of diversity, (ii) the detection of mode collapse, and (iii) the evaluation of generative models for text, image, and graph data.
  arXiv  Detail & Related papers  (2023-11-27T18:19:07Z)
• Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data [68.62134204367668]
  This paper studies score approximation, estimation, and distribution recovery of diffusion models, when data are supported on an unknown low-dimensional linear subspace. We show that with a properly chosen neural network architecture, the score function can be both accurately approximated and efficiently estimated. The generated distribution based on the estimated score function captures the data geometric structures and converges to a close vicinity of the data distribution.
  arXiv  Detail & Related papers  (2023-02-14T17:02:35Z)
• Estimating Divergences in High Dimensions [6.172809837529207]
  We propose the use of decomposable models for estimating divergences in high dimensional data. These allow us to factorize the estimated density of the high-dimensional distribution into a product of lower dimensional functions. We show empirically that estimating the Kullback-Leibler divergence using decomposable models from a maximum likelihood estimator outperforms existing methods for divergence estimation.
  arXiv  Detail & Related papers  (2021-12-08T20:37:28Z)
• Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty [58.144520501201995]
  Bi-Lipschitz regularization of neural network layers preserve relative distances between data instances in the feature spaces of each layer. With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices. We also propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution.
  arXiv  Detail & Related papers  (2021-10-12T22:04:19Z)
• Manifold Hypothesis in Data Analysis: Double Geometrically-Probabilistic Approach to Manifold Dimension Estimation [92.81218653234669]
  We present new approach to manifold hypothesis checking and underlying manifold dimension estimation. Our geometrical method is a modification for sparse data of a well-known box-counting algorithm for Minkowski dimension calculation. Experiments on real datasets show that the suggested approach based on two methods combination is powerful and effective.
  arXiv  Detail & Related papers  (2021-07-08T15:35:54Z)
• Featurized Density Ratio Estimation [82.40706152910292]
  In our work, we propose to leverage an invertible generative model to map the two distributions into a common feature space prior to estimation. This featurization brings the densities closer together in latent space, sidestepping pathological scenarios where the learned density ratios in input space can be arbitrarily inaccurate. At the same time, the invertibility of our feature map guarantees that the ratios computed in feature space are equivalent to those in input space.
  arXiv  Detail & Related papers  (2021-07-05T18:30:26Z)
• Latent Space Model for Higher-order Networks and Generalized Tensor Decomposition [18.07071669486882]
  We introduce a unified framework, formulated as general latent space models, to study complex higher-order network interactions. We formulate the relationship between the latent positions and the observed data via a generalized multilinear kernel as the link function. We demonstrate the effectiveness of our method on synthetic data.
  arXiv  Detail & Related papers  (2021-06-30T13:11:17Z)
• A prior-based approximate latent Riemannian metric [3.716965622352967]
  We propose a surrogate conformal generative metric in the latent space of a generative model that is simple, efficient and robust. We theoretically analyze the behavior of the proposed metric and show that it is sensible to use in practice. Also, we show the applicability of the proposed methodology for data analysis in the life sciences.
  arXiv  Detail & Related papers  (2021-03-09T08:31:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.