Too Good To Be True: performance overestimation in (re)current practices for Human Activity Recognition

• URL: http://arxiv.org/abs/2310.11950v1
• Date: Wed, 18 Oct 2023 13:24:05 GMT
• Title: Too Good To Be True: performance overestimation in (re)current practices for Human Activity Recognition
• Authors: Andr\'es Tello, Victoria Degeler and Alexander Lazovik
• Abstract summary: sliding windows for data segmentation followed by standard random k-fold cross validation produce biased results. It is important to raise awareness in the scientific community about this problem, whose negative effects are being overlooked. Several experiments with different types of datasets and different types of classification models allow us to exhibit the problem and show it persists independently of the method or dataset.
• Score: 49.1574468325115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Today, there are standard and well established procedures within the Human Activity Recognition (HAR) pipeline. However, some of these conventional approaches lead to accuracy overestimation. In particular, sliding windows for data segmentation followed by standard random k-fold cross validation, produce biased results. An analysis of previous literature and present-day studies, surprisingly, shows that these are common approaches in state-of-the-art studies on HAR. It is important to raise awareness in the scientific community about this problem, whose negative effects are being overlooked. Otherwise, publications of biased results lead to papers that report lower accuracies, with correct unbiased methods, harder to publish. Several experiments with different types of datasets and different types of classification models allow us to exhibit the problem and show it persists independently of the method or dataset.

Related papers

• Causality and Independence Enhancement for Biased Node Classification [56.38828085943763]
  We propose a novel Causality and Independence Enhancement (CIE) framework, applicable to various graph neural networks (GNNs) Our approach estimates causal and spurious features at the node representation level and mitigates the influence of spurious correlations. Our approach CIE not only significantly enhances the performance of GNNs but outperforms state-of-the-art debiased node classification methods.
  arXiv  Detail & Related papers  (2023-10-14T13:56:24Z)
• A Systematic Study on Quantifying Bias in GAN-Augmented Data [0.0]
  Generative adversarial networks (GANs) have recently become a popular data augmentation technique used by machine learning practitioners. They have been shown to suffer from the so-called mode collapse failure mode, which makes them vulnerable to exacerbating biases on already skewed datasets. This study is a systematic effort focused on the evaluation of state-of-the-art metrics that can potentially quantify biases in GAN-augmented data.
  arXiv  Detail & Related papers  (2023-08-23T22:19:48Z)
• Targeted Data Augmentation for bias mitigation [0.0]
  We introduce a novel and efficient approach for addressing biases called Targeted Data Augmentation (TDA) Unlike the laborious task of removing biases, our method proposes to insert biases instead, resulting in improved performance. To identify biases, we annotated two diverse datasets: a dataset of clinical skin lesions and a dataset of male and female faces.
  arXiv  Detail & Related papers  (2023-08-22T12:25:49Z)
• Systematic Evaluation of Predictive Fairness [60.0947291284978]
  Mitigating bias in training on biased datasets is an important open problem. We examine the performance of various debiasing methods across multiple tasks. We find that data conditions have a strong influence on relative model performance.
  arXiv  Detail & Related papers  (2022-10-17T05:40:13Z)
• On Modality Bias Recognition and Reduction [70.69194431713825]
  We study the modality bias problem in the context of multi-modal classification. We propose a plug-and-play loss function method, whereby the feature space for each label is adaptively learned. Our method yields remarkable performance improvements compared with the baselines.
  arXiv  Detail & Related papers  (2022-02-25T13:47:09Z)
• ACP++: Action Co-occurrence Priors for Human-Object Interaction Detection [102.9428507180728]
  A common problem in the task of human-object interaction (HOI) detection is that numerous HOI classes have only a small number of labeled examples. We observe that there exist natural correlations and anti-correlations among human-object interactions. We present techniques to learn these priors and leverage them for more effective training, especially on rare classes.
  arXiv  Detail & Related papers  (2021-09-09T06:02:50Z)
• Don't Discard All the Biased Instances: Investigating a Core Assumption in Dataset Bias Mitigation Techniques [19.252319300590656]
  Existing techniques for mitigating dataset bias often leverage a biased model to identify biased instances. The role of these biased instances is then reduced during the training of the main model to enhance its robustness to out-of-distribution data. In this paper, we show that this assumption does not hold in general.
  arXiv  Detail & Related papers  (2021-09-01T10:25:46Z)
• Detecting Human-Object Interactions with Action Co-occurrence Priors [108.31956827512376]
  A common problem in human-object interaction (HOI) detection task is that numerous HOI classes have only a small number of labeled examples. We observe that there exist natural correlations and anti-correlations among human-object interactions. We present techniques to learn these priors and leverage them for more effective training, especially in rare classes.
  arXiv  Detail & Related papers  (2020-07-17T02:47:45Z)
• History-based Anomaly Detector: an Adversarial Approach to Anomaly Detection [3.908842679355254]
  Anomaly detection is a difficult problem in many areas and has recently been subject to a lot of attention. We propose a simple yet new adversarial method to tackle this problem, denoted as History-based anomaly detector (HistoryAD) It consists of a self-supervised model, trained to recognize 'normal' samples by comparing them to samples based on the training history of a previously trained GAN.
  arXiv  Detail & Related papers  (2019-12-26T11:41:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.