Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation -- Extended Version

• URL: http://arxiv.org/abs/2501.09591v1
• Date: Thu, 16 Jan 2025 15:17:27 GMT
• Title: Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation -- Extended Version
• Authors: Muhammad Rajabinasab, Anton D. Lautrup, Arthur Zimek,
• Abstract summary: Existing methods for measuring inter-dataset similarity are computationally expensive, limited, or sensitive to different entities.<n>We propose two novel metrics for measuring inter-dataset similarity.
• Score: 1.6863735232819916
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Measuring inter-dataset similarity is an important task in machine learning and data mining with various use cases and applications. Existing methods for measuring inter-dataset similarity are computationally expensive, limited, or sensitive to different entities and non-trivial choices for parameters. They also lack a holistic perspective on the entire dataset. In this paper, we propose two novel metrics for measuring inter-dataset similarity. We discuss the mathematical foundation and the theoretical basis of our proposed metrics. We demonstrate the effectiveness of the proposed metrics by investigating two applications in the evaluation of synthetic data and in the evaluation of feature selection methods. The theoretical and empirical studies conducted in this paper illustrate the effectiveness of the proposed metrics.

Related papers

• Benchmarking Synthetic Tabular Data: A Multi-Dimensional Evaluation Framework [0.4874819476581695]
  evaluating the quality of synthetic data remains a key challenge for ensuring privacy and utility in data-driven research. We present a framework that quantifies how well synthetic data replicates original distributional properties while ensuring privacy.
  arXiv  Detail & Related papers  (2025-04-02T17:10:30Z)
• Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric [48.81957145701228]
  We propose a new diversity metric based on sample-level "novelty" We show that NovelSum accurately captures diversity variations and achieves a 0.97 correlation with instruction-tuned model performance.
  arXiv  Detail & Related papers  (2025-02-24T14:20:22Z)
• Metrics Revolutions: Groundbreaking Insights into the Implementation of Metrics for Biomedical Image Segmentation [0.0]
  We compare 11 open-source tools for distance-based metrics against our highly accurate mesh-based reference implementation. Results revealed statistically significant differences among all open-source tools are both surprising and concerning.
  arXiv  Detail & Related papers  (2024-10-03T16:14:22Z)
• Enriching Disentanglement: From Logical Definitions to Quantitative Metrics [59.12308034729482]
  Disentangling the explanatory factors in complex data is a promising approach for data-efficient representation learning. We establish relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metrics. We empirically demonstrate the effectiveness of the proposed metrics by isolating different aspects of disentangled representations.
  arXiv  Detail & Related papers  (2023-05-19T08:22:23Z)
• Variable Importance Matching for Causal Inference [73.25504313552516]
  We describe a general framework called Model-to-Match that achieves these goals. Model-to-Match uses variable importance measurements to construct a distance metric. We operationalize the Model-to-Match framework with LASSO.
  arXiv  Detail & Related papers  (2023-02-23T00:43:03Z)
• Measuring Data [79.89948814583805]
  We identify the task of measuring data to quantitatively characterize the composition of machine learning data and datasets. Data measurements quantify different attributes of data along common dimensions that support comparison. We conclude with a discussion of the many avenues of future work, the limitations of data measurements, and how to leverage these measurement approaches in research and practice.
  arXiv  Detail & Related papers  (2022-12-09T22:10:46Z)
• On the role of benchmarking data sets and simulations in method comparison studies [0.0]
  This paper investigates differences and similarities between simulation studies and benchmarking studies. We borrow ideas from different contexts such as mixed methods research and Clinical Scenario Evaluation.
  arXiv  Detail & Related papers  (2022-08-02T13:47:53Z)
• Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics [58.50754318846996]
  In this paper, we show that the performances of metrics are sensitive to data. The ranking of metrics varies when the evaluation is conducted on different datasets.
  arXiv  Detail & Related papers  (2022-03-29T18:58:28Z)
• Learning Personalized Item-to-Item Recommendation Metric via Implicit Feedback [24.37151414523712]
  This paper studies the item-to-item recommendation problem in recommender systems from a new perspective of metric learning via implicit feedback. We develop and investigate a personalizable deep metric model that captures both the internal contents of items and how they were interacted with by users.
  arXiv  Detail & Related papers  (2022-03-18T18:08:57Z)
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories [61.3299263929289]
  Simulation-based inference enables learning the parameters of a model even when its likelihood cannot be computed in practice. One class of methods uses data simulated with different parameters to infer an amortized estimator for the likelihood-to-evidence ratio. We show that this approach can be formulated in terms of mutual information between model parameters and simulated data.
  arXiv  Detail & Related papers  (2021-06-03T12:59:16Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)
• Learning Similarity Metrics for Numerical Simulations [29.39625644221578]
  We propose a neural network-based approach that computes a stable and generalizing metric (LSiM) to compare data from a variety of numerical simulation sources. Our method employs a Siamese network architecture that is motivated by the mathematical properties of a metric.
  arXiv  Detail & Related papers  (2020-02-18T20:11:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.