Reporting Eye-Tracking Data Quality: Towards a New Standard

• URL: http://arxiv.org/abs/2404.00620v1
• Date: Sun, 31 Mar 2024 09:17:34 GMT
• Title: Reporting Eye-Tracking Data Quality: Towards a New Standard
• Authors: Deborah N. Jakobi, Daniel G. Krakowczyk, Lena A. Jäger,
• Abstract summary: This work advocates a new approach of sharing eye-tracking data. Instead of publishing filtered and pre-processed datasets, the eye-tracking data at all pre-processing stages should be published together with data quality reports. In order to transparently report data quality and enable cross-dataset comparisons, we develop data quality reporting standards and metrics that can be automatically applied to a dataset, and integrate them into the open-source Python package pymovements.
• Score: 1.2958449178903728
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Eye-tracking datasets are often shared in the format used by their creators for their original analyses, usually resulting in the exclusion of data considered irrelevant to the primary purpose. In order to increase re-usability of existing eye-tracking datasets for more diverse and initially not considered use cases, this work advocates a new approach of sharing eye-tracking data. Instead of publishing filtered and pre-processed datasets, the eye-tracking data at all pre-processing stages should be published together with data quality reports. In order to transparently report data quality and enable cross-dataset comparisons, we develop data quality reporting standards and metrics that can be automatically applied to a dataset, and integrate them into the open-source Python package pymovements (https://github.com/aeye-lab/pymovements).

Related papers

• Eye-Tracking-while-Reading: A Living Survey of Datasets with Open Library Support [5.162965495020878]
  Eye-tracking-while-reading corpora are a valuable resource for many different disciplines.<n>We aim at creating more transparency and clarity with regards to existing datasets.
  arXiv  Detail & Related papers  (2026-02-23T08:40:50Z)
• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value [74.80873109856563]
  OpenDataArena (ODA) is a holistic and open platform designed to benchmark the intrinsic value of post-training data.<n>ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; and (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources.
  arXiv  Detail & Related papers  (2025-12-16T03:33:24Z)
• DataMIL: Selecting Data for Robot Imitation Learning with Datamodels [77.48472034791213]
  We introduce DataMIL, a policy-driven data selection framework built on the datamodels paradigm.<n>Unlike standard practices that filter data using human notions of quality, DataMIL directly optimize data selection for task success.<n>We validate our approach on a suite of more than 60 simulation and real-world manipulation tasks.
  arXiv  Detail & Related papers  (2025-05-14T17:55:10Z)
• Making Sense of Data in the Wild: Data Analysis Automation at Scale [0.1747623282473278]
  We propose a novel approach that combines intelligent agents with retrieval augmented generation to automate data analysis, dataset curation and indexing at scale. We demonstrate that our approach results in more detailed dataset descriptions, higher hit rates and greater diversity in dataset retrieval tasks.
  arXiv  Detail & Related papers  (2025-01-27T10:04:10Z)
• Diffusion Models as Data Mining Tools [87.77999285241219]
  This paper demonstrates how to use generative models trained for image synthesis as tools for visual data mining. We show that after finetuning conditional diffusion models to synthesize images from a specific dataset, we can use these models to define a typicality measure. This measure assesses how typical visual elements are for different data labels, such as geographic location, time stamps, semantic labels, or even the presence of a disease.
  arXiv  Detail & Related papers  (2024-07-20T17:14:31Z)
• Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective [71.45945607871715]
  We propose Tabular data Pre-Training via Meta-representation (TabPTM) The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels. Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.
  arXiv  Detail & Related papers  (2023-10-31T18:03:54Z)
• Benchmarking and Analyzing Generative Data for Visual Recognition [66.55174903469722]
  This work delves into the impact of generative images, primarily comparing paradigms that harness external data. We devise textbfGenBench, a benchmark comprising 22 datasets with 2548 categories, to appraise generative data across various visual recognition tasks. Our exhaustive benchmark and analysis spotlight generative data's promise in visual recognition, while identifying key challenges for future investigation.
  arXiv  Detail & Related papers  (2023-07-25T17:59:59Z)
• Revisiting Table Detection Datasets for Visually Rich Documents [17.846536373106268]
  This study revisits some open datasets with high-quality annotations, identifies and cleans the noise, and aligns the annotation definitions of these datasets to merge a larger dataset, termed Open-Tables. To enrich the data sources, we propose a new ICT-TD dataset using the PDF files of Information and Communication Technologies (ICT) commodities, a different domain containing unique samples that hardly appear in open datasets. Our experimental results show that the domain differences among existing open datasets are minor despite having different data sources.
  arXiv  Detail & Related papers  (2023-05-04T01:08:15Z)
• Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets [27.562256973255728]
  Natural language processing models often exploit spurious correlations between task-independent features and labels in datasets to perform well only within the distributions they are trained on. We propose to tackle this problem by generating a debiased version of a dataset, which can then be used to train a debiased, off-the-shelf model. Our approach consists of 1) a method for training data generators to generate high-quality, label-consistent data samples; and 2) a filtering mechanism for removing data points that contribute to spurious correlations.
  arXiv  Detail & Related papers  (2022-03-24T09:08:05Z)
• Towards A Reliable Ground-Truth For Biased Language Detection [3.2202224129197745]
  Existing methods to detect bias mostly rely on annotated data to train machine learning models. We evaluate data collection options and compare labels obtained from two popular crowdsourcing platforms. We conclude that detailed annotator training increases data quality, improving the performance of existing bias detection systems.
  arXiv  Detail & Related papers  (2021-12-14T14:13:05Z)
• Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans [103.92680099373567]
  This paper introduces a pipeline to parametrically sample and render multi-task vision datasets from comprehensive 3D scans from the real world. Changing the sampling parameters allows one to "steer" the generated datasets to emphasize specific information. Common architectures trained on a generated starter dataset reached state-of-the-art performance on multiple common vision tasks and benchmarks.
  arXiv  Detail & Related papers  (2021-10-11T04:21:46Z)
• Simple multi-dataset detection [83.9604523643406]
  We present a simple method for training a unified detector on multiple large-scale datasets. We show how to automatically integrate dataset-specific outputs into a common semantic taxonomy. Our approach does not require manual taxonomy reconciliation.
  arXiv  Detail & Related papers  (2021-02-25T18:55:58Z)
• A Note on Data Biases in Generative Models [16.86600007830682]
  We investigate the impact of dataset quality on the performance of generative models. We show how societal biases of datasets are replicated by generative models. We present creative applications through unpaired transfer between diverse datasets such as photographs, oil portraits, and animes.
  arXiv  Detail & Related papers  (2020-12-04T10:46:37Z)
• Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles [66.15398165275926]
  We propose a method that can automatically detect and ignore dataset-specific patterns, which we call dataset biases. Our method trains a lower capacity model in an ensemble with a higher capacity model. We show improvement in all settings, including a 10 point gain on the visual question answering dataset.
  arXiv  Detail & Related papers  (2020-11-07T22:20:03Z)
• Towards End-to-end Video-based Eye-Tracking [50.0630362419371]
  Estimating eye-gaze from images alone is a challenging task due to un-observable person-specific factors. We propose a novel dataset and accompanying method which aims to explicitly learn these semantic and temporal relationships. We demonstrate that the fusion of information from visual stimuli as well as eye images can lead towards achieving performance similar to literature-reported figures.
  arXiv  Detail & Related papers  (2020-07-26T12:39:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.