A Matter of Annotation: An Empirical Study on In Situ and Self-Recall Activity Annotations from Wearable Sensors

• URL: http://arxiv.org/abs/2305.08752v3
• Date: Thu, 11 Jul 2024 13:23:32 GMT
• Title: A Matter of Annotation: An Empirical Study on In Situ and Self-Recall Activity Annotations from Wearable Sensors
• Authors: Alexander Hoelzemann, Kristof Van Laerhoven,
• Abstract summary: We present an empirical study that evaluates and contrasts four commonly employed annotation methods in user studies focused on in-the-wild data collection. For both the user-driven, in situ annotations, where participants annotate their activities during the actual recording process, and the recall methods, where participants retrospectively annotate their data at the end of each day, the participants had the flexibility to select their own set of activity classes and corresponding labels.
• Score: 56.554277096170246
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Research into the detection of human activities from wearable sensors is a highly active field, benefiting numerous applications, from ambulatory monitoring of healthcare patients via fitness coaching to streamlining manual work processes. We present an empirical study that evaluates and contrasts four commonly employed annotation methods in user studies focused on in-the-wild data collection. For both the user-driven, in situ annotations, where participants annotate their activities during the actual recording process, and the recall methods, where participants retrospectively annotate their data at the end of each day, the participants had the flexibility to select their own set of activity classes and corresponding labels. Our study illustrates that different labeling methodologies directly impact the annotations' quality, as well as the capabilities of a deep learning classifier trained with the data. We noticed that in situ methods produce less but more precise labels than recall methods. Furthermore, we combined an activity diary with a visualization tool that enables the participant to inspect and label their activity data. Due to the introduction of such a tool were able to decrease missing annotations and increase the annotation consistency, and therefore the F1-Score of the deep learning model by up to 8% (ranging between 82.1 and 90.4% F1-Score). Furthermore, we discuss the advantages and disadvantages of the methods compared in our study, the biases they could introduce, and the consequences of their usage on human activity recognition studies as well as possible solutions.

Related papers

• Consistency Based Weakly Self-Supervised Learning for Human Activity Recognition with Wearables [1.565361244756411]
  We describe a weakly self-supervised approach for recognizing human activities from sensor-based data. We show that our approach can help the clustering algorithm achieve comparable performance in identifying and categorizing the underlying human activities.
  arXiv  Detail & Related papers  (2024-07-29T06:29:21Z)
• Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective [68.20531518525273]
  We take a closer look into existing self-supervised methods of speech from an information-theoretic perspective. We use linear probes to estimate the mutual information between the target information and learned representations. We explore the potential of evaluating representations in a self-supervised fashion, where we estimate the mutual information between different parts of the data without using any labels.
  arXiv  Detail & Related papers  (2024-01-16T21:13:22Z)
• Unsupervised Embedding Learning for Human Activity Recognition Using Wearable Sensor Data [2.398608007786179]
  We present an unsupervised approach to project the human activities into an embedding space in which similar activities will be located closely together. Results of experiments on three labeled benchmark datasets demonstrate the effectiveness of the framework.
  arXiv  Detail & Related papers  (2023-07-21T08:52:47Z)
• Responsible Active Learning via Human-in-the-loop Peer Study [88.01358655203441]
  We propose a responsible active learning method, namely Peer Study Learning (PSL), to simultaneously preserve data privacy and improve model stability. We first introduce a human-in-the-loop teacher-student architecture to isolate unlabelled data from the task learner (teacher) on the cloud-side. During training, the task learner instructs the light-weight active learner which then provides feedback on the active sampling criterion.
  arXiv  Detail & Related papers  (2022-11-24T13:18:27Z)
• ALLSH: Active Learning Guided by Local Sensitivity and Hardness [98.61023158378407]
  We propose to retrieve unlabeled samples with a local sensitivity and hardness-aware acquisition function. Our method achieves consistent gains over the commonly used active learning strategies in various classification tasks.
  arXiv  Detail & Related papers  (2022-05-10T15:39:11Z)
• Reducing Label Effort: Self-Supervised meets Active Learning [32.4747118398236]
  Recent developments in self-training have achieved very impressive results rivaling supervised learning on some datasets. Our experiments reveal that self-training is remarkably more efficient than active learning at reducing the labeling effort. The performance gap between active learning trained either with self-training or from scratch diminishes as we approach to the point where almost half of the dataset is labeled.
  arXiv  Detail & Related papers  (2021-08-25T20:04:44Z)
• Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering [71.15403434929915]
  We show that across 5 models and 4 datasets on the task of visual question answering, a wide variety of active learning approaches fail to outperform random selection. We identify the problem as collective outliers -- groups of examples that active learning methods prefer to acquire but models fail to learn. We show that active learning sample efficiency increases significantly as the number of collective outliers in the active learning pool decreases.
  arXiv  Detail & Related papers  (2021-07-06T00:52:11Z)
• Confident Coreset for Active Learning in Medical Image Analysis [57.436224561482966]
  We propose a novel active learning method, confident coreset, which considers both uncertainty and distribution for effectively selecting informative samples. By comparative experiments on two medical image analysis tasks, we show that our method outperforms other active learning methods.
  arXiv  Detail & Related papers  (2020-04-05T13:46:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.