Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation -- Extended Version

• URL: http://arxiv.org/abs/2501.09591v1
• Date: Thu, 16 Jan 2025 15:17:27 GMT
• Title: Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation -- Extended Version
• Authors: Muhammad Rajabinasab, Anton D. Lautrup, Arthur Zimek,
• Abstract summary: Existing methods for measuring inter-dataset similarity are computationally expensive, limited, or sensitive to different entities. We propose two novel metrics for measuring inter-dataset similarity.
• Score: 1.6863735232819916
• License:
• Abstract: Measuring inter-dataset similarity is an important task in machine learning and data mining with various use cases and applications. Existing methods for measuring inter-dataset similarity are computationally expensive, limited, or sensitive to different entities and non-trivial choices for parameters. They also lack a holistic perspective on the entire dataset. In this paper, we propose two novel metrics for measuring inter-dataset similarity. We discuss the mathematical foundation and the theoretical basis of our proposed metrics. We demonstrate the effectiveness of the proposed metrics by investigating two applications in the evaluation of synthetic data and in the evaluation of feature selection methods. The theoretical and empirical studies conducted in this paper illustrate the effectiveness of the proposed metrics.

Related papers

• Metrics Revolutions: Groundbreaking Insights into the Implementation of Metrics for Biomedical Image Segmentation [0.0]
  We compare 11 open-source tools for distance-based metrics against our highly accurate mesh-based reference implementation. Results revealed statistically significant differences among all open-source tools are both surprising and concerning.
  arXiv  Detail & Related papers  (2024-10-03T16:14:22Z)
• Measuring Data [79.89948814583805]
  We identify the task of measuring data to quantitatively characterize the composition of machine learning data and datasets. Data measurements quantify different attributes of data along common dimensions that support comparison. We conclude with a discussion of the many avenues of future work, the limitations of data measurements, and how to leverage these measurement approaches in research and practice.
  arXiv  Detail & Related papers  (2022-12-09T22:10:46Z)
• On the role of benchmarking data sets and simulations in method comparison studies [0.0]
  This paper investigates differences and similarities between simulation studies and benchmarking studies. We borrow ideas from different contexts such as mixed methods research and Clinical Scenario Evaluation.
  arXiv  Detail & Related papers  (2022-08-02T13:47:53Z)
• Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics [58.50754318846996]
  In this paper, we show that the performances of metrics are sensitive to data. The ranking of metrics varies when the evaluation is conducted on different datasets.
  arXiv  Detail & Related papers  (2022-03-29T18:58:28Z)
• Learning Personalized Item-to-Item Recommendation Metric via Implicit Feedback [24.37151414523712]
  This paper studies the item-to-item recommendation problem in recommender systems from a new perspective of metric learning via implicit feedback. We develop and investigate a personalizable deep metric model that captures both the internal contents of items and how they were interacted with by users.
  arXiv  Detail & Related papers  (2022-03-18T18:08:57Z)
• Synthetic Benchmarks for Scientific Research in Explainable Machine Learning [14.172740234933215]
  We release XAI-Bench: a suite of synthetic datasets and a library for benchmarking feature attribution algorithms. Unlike real-world datasets, synthetic datasets allow the efficient computation of conditional expected values. We demonstrate the power of our library by benchmarking popular explainability techniques across several evaluation metrics and identifying failure modes for popular explainers.
  arXiv  Detail & Related papers  (2021-06-23T17:10:21Z)
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories [61.3299263929289]
  Simulation-based inference enables learning the parameters of a model even when its likelihood cannot be computed in practice. One class of methods uses data simulated with different parameters to infer an amortized estimator for the likelihood-to-evidence ratio. We show that this approach can be formulated in terms of mutual information between model parameters and simulated data.
  arXiv  Detail & Related papers  (2021-06-03T12:59:16Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• Estimating informativeness of samples with Smooth Unique Information [108.25192785062367]
  We measure how much a sample informs the final weights and how much it informs the function computed by the weights. We give efficient approximations of these quantities using a linearized network. We apply these measures to several problems, such as dataset summarization.
  arXiv  Detail & Related papers  (2021-01-17T10:29:29Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)
• Learning Similarity Metrics for Numerical Simulations [29.39625644221578]
  We propose a neural network-based approach that computes a stable and generalizing metric (LSiM) to compare data from a variety of numerical simulation sources. Our method employs a Siamese network architecture that is motivated by the mathematical properties of a metric.
  arXiv  Detail & Related papers  (2020-02-18T20:11:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.