A Comparative Study of Label-free Representation Quality Metrics in Deep Learning

• URL: http://arxiv.org/abs/2608.23182v1
• Date: Mon, 24 Aug 2026 12:31:37 GMT
• Title: A Comparative Study of Label-free Representation Quality Metrics in Deep Learning
• Abstract summary: We group existing label-free metrics into three families based on their construction.<n>We then characterise the sensitivity of spectral metrics through controlled synthetic experiments.<n>All label-free metrics are evaluated against downstream task accuracy across a diverse set of 260 vision models.
• Score: 1.3572917264310116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a comparative study of label-free metrics for assessing the quality of representations in deep neural networks to understand their reliability under a wide variety of configurations. We group existing label-free metrics into three families based on their construction and analytically establish connections between metrics within the same family. We then characterise the sensitivity of spectral metrics through controlled synthetic experiments. Finally, all label-free metrics are evaluated against downstream task accuracy across a diverse set of 260 vision models on six datasets spanning generic object classification, fine-grained object classification, scene recognition and geospatial task, stratifying results by architecture class and training objective. We find that intrinsic dimensionality (ID) is the most reliable predictor among the metrics considered. However, the reliability of all metrics, including ID, is moderated by architecture class and training objective. Our results provide a clearer understanding of what label-free representation quality metrics measure, when they are reliable, and how to interpret them in practice.

Related papers

• KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text [6.373904244962351]
  We propose a novel, interpretable metric for intrinsic Knowledge Graph quality assessment.<n>It measures how closely an automatically extracted graph approximates an "ideal" graph.<n>We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets.
  arXiv  Detail & Related papers  (2026-07-11T08:45:15Z)
• Do Metrics for Counterfactual Explanations Align with User Perception? [3.919854840656461]
  We compare algorithmic evaluation metrics with human judgments of explanation quality.<n>Our results show that correlations between algorithmic metrics and human ratings are generally weak and strongly dataset-dependent.<n>Our findings suggest that widely used counterfactual evaluation metrics fail to reflect key aspects of explanation quality as perceived by users.
  arXiv  Detail & Related papers  (2026-03-16T17:56:54Z)
• Beyond Surrogates: A Quantitative Analysis for Inter-Metric Relationships [91.61796429377041]
  This paper proposes a unified theoretical framework designed to quantify the relationships between metrics.<n>We categorize metrics into different classes to facilitate a comparative analysis across different mathematical forms.<n>We provide a new perspective on identifying the structural asymmetry in regret transfer, enabling the design of evaluation systems that are theoretically guaranteed to align offline improvements with online objectives.
  arXiv  Detail & Related papers  (2026-03-08T15:08:06Z)
• Learning Unified Distance Metric for Heterogeneous Attribute Data Clustering [60.05209293008078]
  Heterogeneous Attribute Reconstruction and Representation (HARR) learning paradigm for cluster analysis.<n>HarR is parameter-free, convergence-guaranteed, and can more effectively self-adapt to different sought number of clusters $k$.
  arXiv  Detail & Related papers  (2026-03-03T08:13:16Z)
• Categorical Data Clustering via Value Order Estimated Distance Metric Learning [53.28598689867732]
  This paper introduces a novel order distance metric learning approach to intuitively represent categorical attribute values.<n>A new joint learning paradigm is developed to alternatively perform clustering and order distance metric learning.<n>The proposed method achieves superior clustering accuracy on categorical and mixed datasets.
  arXiv  Detail & Related papers  (2024-11-19T08:23:25Z)
• A Novel Metric for Measuring Data Quality in Classification Applications (extended version) [0.0]
  We introduce and explain a novel metric to measure data quality. This metric is based on the correlated evolution between the classification performance and the deterioration of data. We provide an interpretation of each criterion and examples of assessment levels.
  arXiv  Detail & Related papers  (2023-12-13T11:20:09Z)
• Enriching Disentanglement: From Logical Definitions to Quantitative Metrics [59.12308034729482]
  Disentangling the explanatory factors in complex data is a promising approach for data-efficient representation learning. We establish relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metrics. We empirically demonstrate the effectiveness of the proposed metrics by isolating different aspects of disentangled representations.
  arXiv  Detail & Related papers  (2023-05-19T08:22:23Z)
• On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations [74.70957445600936]
  Multiple metrics have been introduced to measure fairness in various natural language processing tasks. These metrics can be roughly categorized into two categories: 1) emphextrinsic metrics for evaluating fairness in downstream applications and 2) emphintrinsic metrics for estimating fairness in upstream language representation models.
  arXiv  Detail & Related papers  (2022-03-25T22:17:43Z)
• Self-Supervised Metric Learning in Multi-View Data: A Downstream Task Perspective [2.01243755755303]
  We study how self-supervised metric learning can benefit downstream tasks in the context of multi-view data. We show that the target distance of metric learning satisfies several desired properties for the downstream tasks. Our analysis characterizes the improvement by self-supervised metric learning on four commonly used downstream tasks.
  arXiv  Detail & Related papers  (2021-06-14T02:34:33Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.