Informed Dataset Selection

• URL: http://arxiv.org/abs/2509.26448v1
• Date: Tue, 30 Sep 2025 16:04:51 GMT
• Title: Informed Dataset Selection
• Authors: Abdullah Abbas, Michael Heep, Theodor Sperle,
• Abstract summary: We developed the APS Explorer, a web application that im- plements the Algorithm Performance Space framework for informed dataset selection.<n>The system analyzes 96 datasets using 28 algorithms across three metrics (nDCG, Hit Ratio, Recall) at five K-values.<n>We extend the APS framework with a statistical based classification system that categorizes datasets into five difficulty levels based on quintiles.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The selection of datasets in recommender systems research lacks a systematic methodology. Researchers often select datasets based on popularity rather than empirical suitability. We developed the APS Explorer, a web application that im- plements the Algorithm Performance Space (APS) framework for informed dataset selection. The system analyzes 96 datasets using 28 algorithms across three metrics (nDCG, Hit Ratio, Recall) at five K-values. We extend the APS framework with a statistical based classification system that categorizes datasets into five difficulty levels based on quintiles. We also introduce a variance-normalized distance metric based on Mahalanobis distance to measure similarity. The APS Explorer was successfully developed with three interactive modules for visualizing algorithm performance, direct comparing algorithms, and analyzing dataset metadata. This tool shifts the process of selecting datasets from intuition-based to evidence-based practices, and it is publicly available at datasets.recommender-systems.com.

Related papers

• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value [74.80873109856563]
  OpenDataArena (ODA) is a holistic and open platform designed to benchmark the intrinsic value of post-training data.<n>ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; and (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources.
  arXiv  Detail & Related papers  (2025-12-16T03:33:24Z)
• APS Explorer: Navigating Algorithm Performance Spaces for Informed Dataset Selection [0.046180371154032895]
  86% of ACM RecSys 2024 papers provide no justification for their dataset choices.<n>Most relying on just four datasets: Amazon (38%), MovieLens (34%), Yelp (15%), and Gowalla (12%)
  arXiv  Detail & Related papers  (2025-08-26T19:46:29Z)
• Adaptive and Robust DBSCAN with Multi-agent Reinforcement Learning [53.527506374566485]
  We propose a novel Adaptive and Robust DBSCAN with Multi-agent Reinforcement Learning cluster framework, namely AR-DBSCAN.<n>We show that AR-DBSCAN not only improves clustering accuracy by up to 144.1% and 175.3% in the NMI and ARI metrics, respectively, but also is capable of robustly finding dominant parameters.
  arXiv  Detail & Related papers  (2025-05-07T11:37:23Z)
• Algorithm Performance Spaces for Strategic Dataset Selection [0.0]
  The evaluation of new algorithms in recommender systems frequently depends on publicly available datasets, such as those from MovieLens or Amazon.<n>This thesis introduces the Algorithm Performance Space, a framework designed to differentiate datasets based on the measured performance of algorithms applied to them.
  arXiv  Detail & Related papers  (2025-04-29T12:29:52Z)
• TSceneJAL: Joint Active Learning of Traffic Scenes for 3D Object Detection [26.059907173437114]
  TSceneJAL framework can efficiently sample the balanced, diverse, and complex traffic scenes from both labeled and unlabeled data.<n>Our approach outperforms existing state-of-the-art methods on 3D object detection tasks with up to 12% improvements.
  arXiv  Detail & Related papers  (2024-12-25T11:07:04Z)
• A Novel Adaptive Fine-Tuning Algorithm for Multimodal Models: Self-Optimizing Classification and Selection of High-Quality Datasets in Remote Sensing [46.603157010223505]
  We propose an adaptive fine-tuning algorithm for multimodal large models. We train the model on two 3090 GPU using one-third of the GeoChat multimodal remote sensing dataset. The model achieved scores of 89.86 and 77.19 on the UCMerced and AID evaluation datasets.
  arXiv  Detail & Related papers  (2024-09-20T09:19:46Z)
• Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models [36.22392593103493]
  Data selection for fine-tuning large language models (LLMs) aims to choose a high-quality subset from existing datasets.<n>Existing surveys overlook an in-depth exploration of the fine-tuning phase.<n>We introduce a novel three-stage scheme - comprising feature extraction, criteria design, and selector evaluation - to systematically categorize and evaluate these methods.
  arXiv  Detail & Related papers  (2024-06-20T08:58:58Z)
• infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information [68.76707843019886]
  infoVerse is a universal framework for dataset characterization. infoVerse captures multidimensional characteristics of datasets by incorporating various model-driven meta-information. In three real-world applications (data pruning, active learning, and data annotation), the samples chosen on infoVerse space consistently outperform strong baselines.
  arXiv  Detail & Related papers  (2023-05-30T18:12:48Z)
• MV-JAR: Masked Voxel Jigsaw and Reconstruction for LiDAR-Based Self-Supervised Pre-Training [58.07391711548269]
  Masked Voxel Jigsaw and Reconstruction (MV-JAR) method for LiDAR-based self-supervised pre-training. Masked Voxel Jigsaw and Reconstruction (MV-JAR) method for LiDAR-based self-supervised pre-training.
  arXiv  Detail & Related papers  (2023-03-23T17:59:02Z)
• Navya3DSeg -- Navya 3D Semantic Segmentation Dataset & split generation for autonomous vehicles [63.20765930558542]
  3D semantic data are useful for core perception tasks such as obstacle detection and ego-vehicle localization. We propose a new dataset, Navya 3D (Navya3DSeg), with a diverse label space corresponding to a large scale production grade operational domain. It contains 23 labeled sequences and 25 supplementary sequences without labels, designed to explore self-supervised and semi-supervised semantic segmentation benchmarks on point clouds.
  arXiv  Detail & Related papers  (2023-02-16T13:41:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.