Quantum-inspired Benchmark for Estimating Intrinsic Dimension

• URL: http://arxiv.org/abs/2510.01335v1
• Date: Wed, 01 Oct 2025 18:03:02 GMT
• Title: Quantum-inspired Benchmark for Estimating Intrinsic Dimension
• Authors: Aritra Das, Joseph T. Iosue, Victor V. Albert,
• Abstract summary: Machine learning models can generalize well on real-world datasets.<n>There exist many methods for ID estimation (IDE) but their estimates vary substantially.<n>We propose a Quantum-Inspired Intrinsic-dimension Estimation (QuIIEst) benchmark.
• Score: 2.0937431058291938
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning models can generalize well on real-world datasets. According to the manifold hypothesis, this is possible because datasets lie on a latent manifold with small intrinsic dimension (ID). There exist many methods for ID estimation (IDE), but their estimates vary substantially. This warrants benchmarking IDE methods on manifolds that are more complex than those in existing benchmarks. We propose a Quantum-Inspired Intrinsic-dimension Estimation (QuIIEst) benchmark consisting of infinite families of topologically non-trivial manifolds with known ID. Our benchmark stems from a quantum-optical method of embedding arbitrary homogeneous spaces while allowing for curvature modification and additive noise. The IDE methods tested were generally less accurate on QuIIEst manifolds than on existing benchmarks under identical resource allocation. We also observe minimal performance degradation with increasingly non-uniform curvature, underscoring the benchmark's inherent difficulty. As a result of independent interest, we perform IDE on the fractal Hofstadter's butterfly and identify which methods are capable of extracting the effective dimension of a space that is not a manifold.

Related papers

• Error Slice Discovery via Manifold Compactness [47.57891946791078]
  There is no proper metric of slice coherence without relying on extra information like predefined slice labels.<n>We propose manifold compactness, a coherence metric without reliance on extra information by incorporating the data geometry property into its design.<n>Then we develop Manifold Compactness based error Slice Discovery (MCSD), a novel algorithm that directly treats risk and coherence as the optimization objective.
  arXiv  Detail & Related papers  (2025-01-31T11:02:07Z)
• Manifold Learning with Sparse Regularised Optimal Transport [1.949927790632678]
  Real-world datasets are subject to noisy observations and sampling, so that distilling information about the underlying manifold is a major challenge.<n>We propose a method for manifold learning that utilises a symmetric version of optimal transport with a quadratic regularisation.<n>We prove that the resulting kernel is consistent with a Laplace-type operator in the continuous limit, establish robustness to heteroskedastic noise and exhibit these results in numerical experiments.
  arXiv  Detail & Related papers  (2023-07-19T08:05:46Z)
• Higher-order topological kernels via quantum computation [68.8204255655161]
  Topological data analysis (TDA) has emerged as a powerful tool for extracting meaningful insights from complex data. We propose a quantum approach to defining Betti kernels, which is based on constructing Betti curves with increasing order.
  arXiv  Detail & Related papers  (2023-07-14T14:48:52Z)
• Intrinsic Gaussian Process on Unknown Manifolds with Probabilistic Metrics [5.582101184758529]
  This article presents a novel approach to construct Intrinsic Gaussian Processes for regression on unknown manifold with probabilistic metrics in point clouds. The geometry of manifold is in general different from the usual Euclidean geometry. The applications of GPUM are illustrated in the simulation studies on the Swiss roll, high dimensional real datasets of WiFi signals and image data examples.
  arXiv  Detail & Related papers  (2023-01-16T17:42:40Z)
• Semi-Supervised Manifold Learning with Complexity Decoupled Chart Autoencoders [45.29194877564103]
  This work introduces a chart autoencoder with an asymmetric encoding-decoding process that can incorporate additional semi-supervised information such as class labels. We discuss the approximation power of such networks and derive a bound that essentially depends on the intrinsic dimension of the data manifold rather than the dimension of ambient space.
  arXiv  Detail & Related papers  (2022-08-22T19:58:03Z)
• Intrinsic dimension estimation for discrete metrics [65.5438227932088]
  In this letter we introduce an algorithm to infer the intrinsic dimension (ID) of datasets embedded in discrete spaces. We demonstrate its accuracy on benchmark datasets, and we apply it to analyze a metagenomic dataset for species fingerprinting. This suggests that evolutive pressure acts on a low-dimensional manifold despite the high-dimensionality of sequences' space.
  arXiv  Detail & Related papers  (2022-07-20T06:38:36Z)
• Measuring dissimilarity with diffeomorphism invariance [94.02751799024684]
  We introduce DID, a pairwise dissimilarity measure applicable to a wide range of data spaces. We prove that DID enjoys properties which make it relevant for theoretical study and practical use.
  arXiv  Detail & Related papers  (2022-02-11T13:51:30Z)
• EGGS: Eigen-Gap Guided Search Making Subspace Clustering Easy [20.547648917833698]
  We present an eigen-gap guided search method for subspace clustering. We show, theoretically and numerically, that the Laplacian matrix with a larger relative-eigen-gap often yields a higher clustering accuracy and stability. Our method has high flexibility and convenience in real applications, and also has low computational cost.
  arXiv  Detail & Related papers  (2021-07-23T08:53:36Z)
• Manifold Topology Divergence: a Framework for Comparing Data Manifolds [109.0784952256104]
  We develop a framework for comparing data manifold, aimed at the evaluation of deep generative models. Based on the Cross-Barcode, we introduce the Manifold Topology Divergence score (MTop-Divergence) We demonstrate that the MTop-Divergence accurately detects various degrees of mode-dropping, intra-mode collapse, mode invention, and image disturbance.
  arXiv  Detail & Related papers  (2021-06-08T00:30:43Z)
• Manifold Learning via Manifold Deflation [105.7418091051558]
  dimensionality reduction methods provide a valuable means to visualize and interpret high-dimensional data. Many popular methods can fail dramatically, even on simple two-dimensional Manifolds. This paper presents an embedding method for a novel, incremental tangent space estimator that incorporates global structure as coordinates. Empirically, we show our algorithm recovers novel and interesting embeddings on real-world and synthetic datasets.
  arXiv  Detail & Related papers  (2020-07-07T10:04:28Z)
• Learning Flat Latent Manifolds with VAEs [16.725880610265378]
  We propose an extension to the framework of variational auto-encoders, where the Euclidean metric is a proxy for the similarity between data points. We replace the compact prior typically used in variational auto-encoders with a recently presented, more expressive hierarchical one. We evaluate our method on a range of data-sets, including a video-tracking benchmark.
  arXiv  Detail & Related papers  (2020-02-12T09:54:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.