Metric Design != Metric Behavior: Improving Metric Selection for the Unbiased Evaluation of Dimensionality Reduction

• URL: http://arxiv.org/abs/2507.02225v1
• Date: Thu, 03 Jul 2025 01:07:02 GMT
• Title: Metric Design != Metric Behavior: Improving Metric Selection for the Unbiased Evaluation of Dimensionality Reduction
• Authors: Jiyeon Bae, Hyeon Jeon, Jinwook Seo,
• Abstract summary: dimensionality reduction (DR) projections are crucial for reliable visual analytics.<n>DR projections can become biased if highly correlated metrics--those measuring similar structural characteristics--are inadvertently selected.<n>We propose a novel workflow that reduces bias in the selection of evaluation metrics by clustering metrics based on their empirical correlations.
• Score: 10.099350224451387
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Evaluating the accuracy of dimensionality reduction (DR) projections in preserving the structure of high-dimensional data is crucial for reliable visual analytics. Diverse evaluation metrics targeting different structural characteristics have thus been developed. However, evaluations of DR projections can become biased if highly correlated metrics--those measuring similar structural characteristics--are inadvertently selected, favoring DR techniques that emphasize those characteristics. To address this issue, we propose a novel workflow that reduces bias in the selection of evaluation metrics by clustering metrics based on their empirical correlations rather than on their intended design characteristics alone. Our workflow works by computing metric similarity using pairwise correlations, clustering metrics to minimize overlap, and selecting a representative metric from each cluster. Quantitative experiments demonstrate that our approach improves the stability of DR evaluation, which indicates that our workflow contributes to mitigating evaluation bias.

Related papers

• Reranking-based Generation for Unbiased Perspective Summarization [10.71668103641552]
  We develop a test set for benchmarking metric reliability using human annotations.<n>We show that traditional metrics underperform compared to language model-based metrics, which prove to be strong evaluators.<n>Our findings aim to contribute to the reliable evaluation and development of perspective summarization methods.
  arXiv  Detail & Related papers  (2025-06-19T00:01:43Z)
• A Unifying Information-theoretic Perspective on Evaluating Generative Models [5.524685807042777]
  Several recent approaches utilize "precision" and "recall," borrowed from the classification domain, to individually quantify the output fidelity (realism) and output diversity (representation of the real data variation)<n>We unify a class of kth-nearest-neighbors (kNN)-based metrics under an information-theoretic lens using approaches from kNN density estimation.<n>We propose a tri-dimensional metric composed of Precision Cross-Entropy (PCE), Recall Cross-Entropy (RCE), and Recall Entropy (RE)
  arXiv  Detail & Related papers  (2024-12-18T21:17:02Z)
• Is Reference Necessary in the Evaluation of NLG Systems? When and Where? [58.52957222172377]
  We show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.
  arXiv  Detail & Related papers  (2024-03-21T10:31:11Z)
• What is the Best Automated Metric for Text to Motion Generation? [19.71712698183703]
  There is growing interest in generating skeleton-based human motions from natural language descriptions. Human evaluation is the ultimate accuracy measure for this task, and automated metrics should correlate well with human quality judgments. This paper systematically studies which metrics best align with human evaluations and proposes new metrics that align even better.
  arXiv  Detail & Related papers  (2023-09-19T01:59:54Z)
• Learning Evaluation Models from Large Language Models for Sequence Generation [61.8421748792555]
  We propose a three-stage evaluation model training method that utilizes large language models to generate labeled data for model-based metric development.<n> Experimental results on the SummEval benchmark demonstrate that CSEM can effectively train an evaluation model without human-labeled data.
  arXiv  Detail & Related papers  (2023-08-08T16:41:16Z)
• An evaluation framework for dimensionality reduction through sectional curvature [59.40521061783166]
  In this work, we aim to introduce the first highly non-supervised dimensionality reduction performance metric. To test its feasibility, this metric has been used to evaluate the performance of the most commonly used dimension reduction algorithms. A new parameterized problem instance generator has been constructed in the form of a function generator.
  arXiv  Detail & Related papers  (2023-03-17T11:59:33Z)
• Never mind the metrics -- what about the uncertainty? Visualising confusion matrix metric distributions [6.566615606042994]
  This paper strives for a more balanced perspective on classifier performance metrics by highlighting their distributions under different models of uncertainty. We develop equations, animations and interactive visualisations of the contours of performance metrics within (and beyond) this ROC space. Our hope is that these insights and visualisations will raise greater awareness of the substantial uncertainty in performance metric estimates.
  arXiv  Detail & Related papers  (2022-06-05T11:54:59Z)
• REAM$\sharp$: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation [63.46331073232526]
  We present an enhancement approach to Reference-based EvAluation Metrics for open-domain dialogue systems. A prediction model is designed to estimate the reliability of the given reference set. We show how its predicted results can be helpful to augment the reference set, and thus improve the reliability of the metric.
  arXiv  Detail & Related papers  (2021-05-30T10:04:13Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• AttriMeter: An Attribute-guided Metric Interpreter for Person Re-Identification [100.3112429685558]
  Person ReID systems only provide a distance or similarity when matching two persons. We propose an Attribute-guided Metric Interpreter, named AttriMeter, to semantically and quantitatively explain the results of CNN-based ReID models.
  arXiv  Detail & Related papers  (2021-03-02T03:37:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.