Yet it moves: Learning from Generic Motions to Generate IMU data from YouTube videos

• URL: http://arxiv.org/abs/2011.11600v1
• Date: Mon, 23 Nov 2020 18:16:46 GMT
• Title: Yet it moves: Learning from Generic Motions to Generate IMU data from YouTube videos
• Authors: Vitor Fortes Rey, Kamalveer Kaur Garewal, Paul Lukowicz
• Abstract summary: We show how we can train a regression model on generic motions for both accelerometer and gyro signals to generate synthetic IMU data. We demonstrate that systems trained on simulated data generated by our regression model can come to within around 10% of the mean F1 score of a system trained on real sensor data.
• Score: 5.008235182488304
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Human activity recognition (HAR) using wearable sensors has benefited much less from recent advances in Machine Learning than fields such as computer vision and natural language processing. This is to a large extent due to the lack of large scale repositories of labeled training data. In our research we aim to facilitate the use of online videos, which exists in ample quantity for most activities and are much easier to label than sensor data, to simulate labeled wearable motion sensor data. In previous work we already demonstrate some preliminary results in this direction focusing on very simple, activity specific simulation models and a single sensor modality (acceleration norm)\cite{10.1145/3341162.3345590}. In this paper we show how we can train a regression model on generic motions for both accelerometer and gyro signals and then apply it to videos of the target activities to generate synthetic IMU data (acceleration and gyro norms) that can be used to train and/or improve HAR models. We demonstrate that systems trained on simulated data generated by our regression model can come to within around 10% of the mean F1 score of a system trained on real sensor data. Furthermore we show that by either including a small amount of real sensor data for model calibration or simply leveraging the fact that (in general) we can easily generate much more simulated data from video than we can collect in terms of real sensor data the advantage of real sensor data can be eventually equalized.

Related papers

• Scaling Wearable Foundation Models [54.93979158708164]
  We investigate the scaling properties of sensor foundation models across compute, data, and model size. Using a dataset of up to 40 million hours of in-situ heart rate, heart rate variability, electrodermal activity, accelerometer, skin temperature, and altimeter per-minute data from over 165,000 people, we create LSM. Our results establish the scaling laws of LSM for tasks such as imputation, extrapolation, both across time and sensor modalities.
  arXiv  Detail & Related papers  (2024-10-17T15:08:21Z)
• Exploring Text-to-Motion Generation with Human Preference [59.28730218998923]
  This paper presents an exploration of preference learning in text-to-motion generation. We find that current improvements in text-to-motion generation still rely on datasets requiring expert labelers with motion capture systems. We show that preference learning has the potential to greatly improve current text-to-motion generative models.
  arXiv  Detail & Related papers  (2024-04-15T04:14:42Z)
• A Plug-in Tiny AI Module for Intelligent and Selective Sensor Data Transmission [10.174575604689391]
  We propose a novel sensing module to equip sensing frameworks with intelligent data transmission capabilities. We integrate a highly efficient machine learning model placed near the sensor. This model provides prompt feedback for the sensing system to transmit only valuable data while discarding irrelevant information.
  arXiv  Detail & Related papers  (2024-02-03T05:41:39Z)
• Any-point Trajectory Modeling for Policy Learning [64.23861308947852]
  We introduce Any-point Trajectory Modeling (ATM) to predict future trajectories of arbitrary points within a video frame. ATM outperforms strong video pre-training baselines by 80% on average. We show effective transfer learning of manipulation skills from human videos and videos from a different robot morphology.
  arXiv  Detail & Related papers  (2023-12-28T23:34:43Z)
• Virtual Fusion with Contrastive Learning for Single Sensor-based Activity Recognition [5.225544155289783]
  Various types of sensors can be used for Human Activity Recognition (HAR) Sometimes a single sensor cannot fully observe the user's motions from its perspective, which causes wrong predictions. We propose Virtual Fusion - a new method that takes advantage of unlabeled data from multiple time-synchronized sensors during training, but only needs one sensor for inference.
  arXiv  Detail & Related papers  (2023-12-01T17:03:27Z)
• Unsupervised Statistical Feature-Guided Diffusion Model for Sensor-based Human Activity Recognition [3.2319909486685354]
  A key problem holding up progress in wearable sensor-based human activity recognition is the unavailability of diverse and labeled training data. We propose an unsupervised statistical feature-guided diffusion model specifically optimized for wearable sensor-based human activity recognition. By conditioning the diffusion model on statistical information such as mean, standard deviation, Z-score, and skewness, we generate diverse and representative synthetic sensor data.
  arXiv  Detail & Related papers  (2023-05-30T15:12:59Z)
• Convolutional Neural Networks for the classification of glitches in gravitational-wave data streams [52.77024349608834]
  We classify transient noise signals (i.e.glitches) and gravitational waves in data from the Advanced LIGO detectors. We use models with a supervised learning approach, both trained from scratch using the Gravity Spy dataset. We also explore a self-supervised approach, pre-training models with automatically generated pseudo-labels.
  arXiv  Detail & Related papers  (2023-03-24T11:12:37Z)
• Learning to Simulate Realistic LiDARs [66.7519667383175]
  We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
  arXiv  Detail & Related papers  (2022-09-22T13:12:54Z)
• Video2IMU: Realistic IMU features and signals from videos [0.7087237546722618]
  Human Activity Recognition (HAR) from wearable sensor data identifies movements or activities in unconstrained environments. We propose the use of neural networks for the generation of realistic signals and features using human activity monocular videos. We show how these generated features and signals can be utilized, instead of their real counterparts, to train HAR models.
  arXiv  Detail & Related papers  (2022-02-14T08:37:26Z)
• Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation [98.51313127382937]
  We focus on the use of labels in the synthetic domain alone. Our approach introduces both a way to learn neural-invariant representations and a theoretically inspired view on how to sample the data from the simulator. We showcase our approach on the bird's-eye-view vehicle segmentation task with multi-sensor data.
  arXiv  Detail & Related papers  (2021-11-15T18:37:43Z)
• IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition [12.91206329972949]
  We introduce IMUTube, an automated processing pipeline to convert videos of human activity into virtual streams of IMU data. These virtual IMU streams represent accelerometry at a wide variety of locations on the human body. We show how the virtually-generated IMU data improves the performance of a variety of models on known HAR datasets.
  arXiv  Detail & Related papers  (2020-05-29T21:50:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.