Beyond Confusion: A Fine-grained Dialectical Examination of Human Activity Recognition Benchmark Datasets

• URL: http://arxiv.org/abs/2412.09037v1
• Date: Thu, 12 Dec 2024 07:53:17 GMT
• Title: Beyond Confusion: A Fine-grained Dialectical Examination of Human Activity Recognition Benchmark Datasets
• Authors: Daniel Geissler, Dominique Nshimyimana, Vitor Fortes Rey, Sungho Suh, Bo Zhou, Paul Lukowicz,
• Abstract summary: The research of machine learning algorithms for human activity recognition (HAR) has made significant progress with publicly available datasets. Recent models like transformers have been applied to HAR datasets with limited success from the benchmark metrics. This paper aims to address these open questions by conducting a fine-grained inspection of six popular HAR benchmark datasets.
• Score: 4.249657064343807
• License:
• Abstract: The research of machine learning (ML) algorithms for human activity recognition (HAR) has made significant progress with publicly available datasets. However, most research prioritizes statistical metrics over examining negative sample details. While recent models like transformers have been applied to HAR datasets with limited success from the benchmark metrics, their counterparts have effectively solved problems on similar levels with near 100% accuracy. This raises questions about the limitations of current approaches. This paper aims to address these open questions by conducting a fine-grained inspection of six popular HAR benchmark datasets. We identified for some parts of the data, none of the six chosen state-of-the-art ML methods can correctly classify, denoted as the intersect of false classifications (IFC). Analysis of the IFC reveals several underlying problems, including ambiguous annotations, irregularities during recording execution, and misaligned transition periods. We contribute to the field by quantifying and characterizing annotated data ambiguities, providing a trinary categorization mask for dataset patching, and stressing potential improvements for future data collections.

Related papers

• A Similarity-Based Oversampling Method for Multi-label Imbalanced Text Data [1.799933345199395]
  This study introduces and examines a novel oversampling method for multi-label text classification. The proposed method identifies potential new samples from unlabeled data by leveraging similarity measures between instances. By iteratively searching the unlabeled dataset, the method locates instances similar to those in underrepresented classes. Instances that demonstrate performance improvement are then added to the labeled dataset.
  arXiv  Detail & Related papers  (2024-11-01T20:33:49Z)
• A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deployment [76.04306818209753]
  We introduce a substantial crowdsourcing annotation dataset collected from a real-world crowdsourcing platform. This dataset comprises approximately two thousand workers, one million tasks, and six million annotations. We evaluate the effectiveness of several representative truth inference algorithms on this dataset.
  arXiv  Detail & Related papers  (2024-03-10T16:00:41Z)
• Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models [25.022166664832596]
  We propose a simple and effective approach to detect data contamination in large language models (LLMs) and estimate the amount of it. We frame data contamination detection as a series of multiple-choice questions and devise a quiz format wherein three perturbed versions of each subsampled instance from a specific dataset partition are created. Our findings suggest that DCQ achieves state-of-the-art results and uncovers greater contamination/memorization levels compared to existing methods.
  arXiv  Detail & Related papers  (2023-11-10T18:48:58Z)
• Binary Quantification and Dataset Shift: An Experimental Investigation [54.14283123210872]
  Quantification is the supervised learning task that consists of training predictors of the class prevalence values of sets of unlabelled data. The relationship between quantification and other types of dataset shift remains, by and large, unexplored. We propose a fine-grained taxonomy of types of dataset shift, by establishing protocols for the generation of datasets affected by these types of shift.
  arXiv  Detail & Related papers  (2023-10-06T20:11:27Z)
• Integrating Transformer and Autoencoder Techniques with Spectral Graph Algorithms for the Prediction of Scarcely Labeled Molecular Data [2.8360662552057323]
  This work introduces three graph-based models incorporating Merriman-Bence-Osher (MBO) techniques to tackle this challenge. Specifically, graph-based modifications of the MBO scheme is integrated with state-of-the-art techniques, including a home-made transformer and an autoencoder. The proposed models are validated using five benchmark data sets.
  arXiv  Detail & Related papers  (2022-11-12T22:45:32Z)
• Multiple Instance Learning for Detecting Anomalies over Sequential Real-World Datasets [2.427831679672374]
  Multiple Instance Learning (MIL) has been shown effective on problems with incomplete knowledge of labels in the training dataset. We propose an MIL-based formulation and various algorithmic instantiations of this framework based on different design decisions. The framework generalizes well over diverse datasets resulting from different real-world application domains.
  arXiv  Detail & Related papers  (2022-10-04T16:02:09Z)
• Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection [85.11649974840758]
  3D object detection networks tend to be biased towards the data they are trained on. We propose a single-frame approach for source-free, unsupervised domain adaptation of lidar-based 3D object detectors.
  arXiv  Detail & Related papers  (2021-11-30T18:42:42Z)
• Doing Great at Estimating CATE? On the Neglected Assumptions in Benchmark Comparisons of Treatment Effect Estimators [91.3755431537592]
  We show that even in arguably the simplest setting, estimation under ignorability assumptions can be misleading. We consider two popular machine learning benchmark datasets for evaluation of heterogeneous treatment effect estimators. We highlight that the inherent characteristics of the benchmark datasets favor some algorithms over others.
  arXiv  Detail & Related papers  (2021-07-28T13:21:27Z)
• Self-Trained One-class Classification for Unsupervised Anomaly Detection [56.35424872736276]
  Anomaly detection (AD) has various applications across domains, from manufacturing to healthcare. In this work, we focus on unsupervised AD problems whose entire training data are unlabeled and may contain both normal and anomalous samples. To tackle this problem, we build a robust one-class classification framework via data refinement. We show that our method outperforms state-of-the-art one-class classification method by 6.3 AUC and 12.5 average precision.
  arXiv  Detail & Related papers  (2021-06-11T01:36:08Z)
• Stance Detection Benchmark: How Robust Is Your Stance Detection? [65.91772010586605]
  Stance Detection (StD) aims to detect an author's stance towards a certain topic or claim. We introduce a StD benchmark that learns from ten StD datasets of various domains in a multi-dataset learning setting. Within this benchmark setup, we are able to present new state-of-the-art results on five of the datasets.
  arXiv  Detail & Related papers  (2020-01-06T13:37:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.