Frequency-Aware Masked Autoencoders for Human Activity Recognition using Accelerometers

• URL: http://arxiv.org/abs/2502.17477v1
• Date: Mon, 17 Feb 2025 14:57:51 GMT
• Title: Frequency-Aware Masked Autoencoders for Human Activity Recognition using Accelerometers
• Authors: Niels R. Lorenzen, Poul J. Jennum, Emmanuel Mignot, Andreas Brink-Kjaer,
• Abstract summary: Supervised machine learning and deep learning algorithms have long been used to extract meaningful activity information from raw accelerometry data.<n>We propose a novel spectrogram-based loss function named the log-scale mean magnitude (LMM) loss for human activity recognition.<n>Our findings demonstrate that the LMM loss is a robust and effective method for pretraining MAE models on accelerometer data for HAR.
• Score: 0.1499944454332829
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Wearable accelerometers are widely used for continuous monitoring of physical activity. Supervised machine learning and deep learning algorithms have long been used to extract meaningful activity information from raw accelerometry data, but progress has been hampered by the limited amount of publicly available labeled data. Exploiting large unlabeled datasets using self-supervised pretraining is a relatively new and underexplored approach in the field of human activity recognition (HAR). We used a time-series transformer masked autoencoder (MAE) approach to self-supervised pretraining and propose a novel spectrogram-based loss function named the log-scale mean magnitude (LMM) loss. We compared MAE models pretrained with LMM to one trained with the mean squared error (MSE) loss. We leveraged the large unlabeled UK Biobank accelerometry dataset (n = 109k) for pretraining and evaluated downstream HAR performance using linear classifier in a smaller labelled dataset. We found that pretraining with the LMM loss improved performance compared to a model pretrained with the MSE loss, with balanced accuracies of 0.848 and 0.709, respectively. Further analysis revealed that better convergence of the LMM loss, but not the MSE loss significantly correlated with improved downstream performance (r=-0.61, p=0.04) for balanced accuracy). Finally, we compared our MAE models to the state-of-the-art for HAR, also pretrained on the UK Biobank accelerometry data. Our LMM-pretrained models performed better when finetuned using a linear classifier and performed comparably when finetuned using an LSTM classifier, while MSE-pretrained models consistently underperformed. Our findings demonstrate that the LMM loss is a robust and effective method for pretraining MAE models on accelerometer data for HAR. Future work should explore optimizing loss function combinations and extending our approach to other tasks.

Related papers

• Targeted Adversarial Denoising Autoencoders (TADA) for Neural Time Series Filtration [0.0]
  Current machine learning (ML)-based algorithms for filtering electroencephalography (EEG) time series data face challenges related to cumbersome training times, regularization, and accurate reconstruction.<n>We present an ML filtration algorithm driven by a logistic covariance-targeted adversarial denoising autoencoder (TADA)<n>To test this hypothesis, a TADA system prototype was trained and evaluated on the task of removing electromyographic (EMG) noise from EEG data in the EEGdenoiseNet dataset.
  arXiv  Detail & Related papers  (2025-01-09T04:41:50Z)
• Attribute-to-Delete: Machine Unlearning via Datamodel Matching [65.13151619119782]
  Machine unlearning -- efficiently removing a small "forget set" training data on a pre-divertrained machine learning model -- has recently attracted interest. Recent research shows that machine unlearning techniques do not hold up in such a challenging setting.
  arXiv  Detail & Related papers  (2024-10-30T17:20:10Z)
• Scaling Laws for Predicting Downstream Performance in LLMs [75.28559015477137]
  This work focuses on the pre-training loss as a more-efficient metric for performance estimation. We extend the power law analytical function to predict domain-specific pre-training loss based on FLOPs across data sources. We employ a two-layer neural network to model the non-linear relationship between multiple domain-specific loss and downstream performance.
  arXiv  Detail & Related papers  (2024-10-11T04:57:48Z)
• A deeper look at depth pruning of LLMs [49.30061112976263]
  Large Language Models (LLMs) are resource-intensive to train but more costly to deploy in production. Recent work has attempted to prune blocks of LLMs based on cheap proxies for estimating block importance. We show that adaptive metrics exhibit a trade-off in performance between tasks.
  arXiv  Detail & Related papers  (2024-07-23T08:40:27Z)
• CE-SSL: Computation-Efficient Semi-Supervised Learning for ECG-based Cardiovascular Diseases Detection [16.34314710823127]
  We propose a computation-efficient semi-supervised learning paradigm (CE-SSL) for robust and computation-efficient CVDs detection using ECG. It enables a robust adaptation of pre-trained models on downstream datasets with limited supervision and high computational efficiency. CE-SSL not only outperforms the state-of-the-art methods in multi-label CVDs detection but also consumes fewer GPU footprints, training time, and parameter storage space.
  arXiv  Detail & Related papers  (2024-06-20T14:45:13Z)
• PUMA: margin-based data pruning [51.12154122266251]
  We focus on data pruning, where some training samples are removed based on the distance to the model classification boundary (i.e., margin) We propose PUMA, a new data pruning strategy that computes the margin using DeepFool. We show that PUMA can be used on top of the current state-of-the-art methodology in robustness, and it is able to significantly improve the model performance unlike the existing data pruning strategies.
  arXiv  Detail & Related papers  (2024-05-10T08:02:20Z)
• Scaling Relationship on Learning Mathematical Reasoning with Large Language Models [75.29595679428105]
  We investigate how the pre-training loss, supervised data amount, and augmented data amount influence the reasoning performances of a supervised LLM. We find that rejection samples from multiple models push LLaMA-7B to an accuracy of 49.3% on GSM8K which outperforms the supervised fine-tuning (SFT) accuracy of 35.9% significantly.
  arXiv  Detail & Related papers  (2023-08-03T15:34:01Z)
• Self-learning locally-optimal hypertuning using maximum entropy, and comparison of machine learning approaches for estimating fatigue life in composite materials [0.0]
  We develop an ML nearest-neighbors-alike algorithm based on the principle of maximum entropy to predict fatigue damage. The predictions achieve a good level of accuracy, similar to other ML algorithms.
  arXiv  Detail & Related papers  (2022-10-19T12:20:07Z)
• A comparison of Monte Carlo dropout and bootstrap aggregation on the performance and uncertainty estimation in radiation therapy dose prediction with deep learning neural networks [0.46180371154032895]
  We propose to use Monte Carlo dropout (MCDO) and the bootstrap aggregation (bagging) technique on deep learning models to produce uncertainty estimations for radiation therapy dose prediction. Performance-wise, bagging provides statistically significant reduced loss value and errors in most of the metrics investigated.
  arXiv  Detail & Related papers  (2020-11-01T00:24:43Z)
• Estimating the time-lapse between medical insurance reimbursement with non-parametric regression models [0.0]
  We comparatively study the properties of four nonparametric algorithms, K-Nearest Neighbours (KNNs), Support Vector Machines (SVMs), Decision trees and Random forests. The supervised learning task is a regression estimate of the time-lapse in medical insurance reimbursement.
  arXiv  Detail & Related papers  (2020-08-19T18:39:12Z)
• Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources [78.72922528736011]
  We propose a novel approach, black-box adversarial reprogramming (BAR), that repurposes a well-trained black-box machine learning model. Using zeroth order optimization and multi-label mapping techniques, BAR can reprogram a black-box ML model solely based on its input-output responses. BAR outperforms state-of-the-art methods and yields comparable performance to the vanilla adversarial reprogramming method.
  arXiv  Detail & Related papers  (2020-07-17T01:52:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.