HARMES: A Multi-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound

• URL: http://arxiv.org/abs/2605.02596v1
• Date: Mon, 04 May 2026 13:47:41 GMT
• Title: HARMES: A Multi-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound
• Abstract summary: This work introduces HARMES, a multi-modal wearable dataset combining three wrist-recorded modalities.<n>HarMES totals over 80 hours of recorded data, with approximately three hours of labeled activity data per participant.<n>To the best of our knowledge, HARMES is the first dataset to combine this particular sensor trio.
• Score: 22.090898322586824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With each sensing modality exhibiting inherent strengths and limitations, multi-modal approaches for wearable Human Activity Recognition (HAR) are becoming increasingly relevant -- particularly for recognizing Activities of Daily Living (ADLs), where individual modalities often produce ambiguous signals for similar or complex activities. This work introduces HARMES, a multi-modal wearable dataset combining three wrist-recorded modalities: motion sensing via an Inertial Measurement Unit (IMU), atmospheric environmental sensors (humidity, temperature, and pressure), and audio. Collected from 20 participants performing household activities in their own homes, HARMES totals over 80 hours of recorded data, with approximately three hours of labeled activity data per participant across 15 ADL classes. To the best of our knowledge, HARMES is the first dataset to combine this particular sensor trio, and it is nearly six times larger than the previously largest wrist-inertial-acoustic HAR dataset. In an extensive benchmark, we evaluate cross-subject generalization and conduct an ablation study revealing that modality contributions are activity-dependent and can provide complementary value, particularly for activities that are ambiguous from motion data alone. HARMES is freely available at Zenodo, alongside example code for loading the dataset and training models on GitHub.

Related papers

• InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation [54.09384502044162]
  We introduce InterAct, a large-scale 3D HOI benchmark featuring dataset and methodological advancements.<n>First, we consolidate and standardize 21.81 hours of HOI data from diverse sources, enriching it with detailed textual annotations.<n>Second, we propose a unified optimization framework to enhance data quality by reducing artifacts and correcting hand motions.<n>Third, we define six benchmarking tasks and develop a unified HOI generative modeling perspective, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-09-11T15:43:54Z)
• Hierarchical and Multimodal Data for Daily Activity Understanding [11.200514097148776]
  Daily Activity Recordings for Artificial Intelligence (DARai) is a multimodal dataset constructed to understand human activities in real-world settings.<n>DARai consists of continuous scripted and unscripted recordings of 50 participants in 10 different environments, totaling over 200 hours of data.<n> Experiments with various machine learning models showcase the value of DARai in uncovering important challenges in human-centered applications.
  arXiv  Detail & Related papers  (2025-04-24T16:04:00Z)
• DISCOVER: Data-driven Identification of Sub-activities via Clustering and Visualization for Enhanced Activity Recognition in Smart Homes [46.86909768552777]
  We introduce DISCOVER, a method to discover fine-grained human sub-activities from unlabeled sensor data without relying on pre-segmentation.<n>We demonstrate its effectiveness through a re-annotation exercise on widely used HAR datasets.
  arXiv  Detail & Related papers  (2025-02-11T20:02:24Z)
• Scaling Wearable Foundation Models [54.93979158708164]
  We investigate the scaling properties of sensor foundation models across compute, data, and model size. Using a dataset of up to 40 million hours of in-situ heart rate, heart rate variability, electrodermal activity, accelerometer, skin temperature, and altimeter per-minute data from over 165,000 people, we create LSM. Our results establish the scaling laws of LSM for tasks such as imputation, extrapolation, both across time and sensor modalities.
  arXiv  Detail & Related papers  (2024-10-17T15:08:21Z)
• MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World [55.878173953175356]
  We propose MultiPLY, a multisensory embodied large language model. We first collect Multisensory Universe, a large-scale multisensory interaction dataset comprising 500k data. We demonstrate that MultiPLY outperforms baselines by a large margin through a diverse set of embodied tasks.
  arXiv  Detail & Related papers  (2024-01-16T18:59:45Z)
• Human-centric Scene Understanding for 3D Large-scale Scenarios [52.12727427303162]
  We present a large-scale multi-modal dataset for human-centric scene understanding, dubbed HuCenLife. Our HuCenLife can benefit many 3D perception tasks, such as segmentation, detection, action recognition, etc.
  arXiv  Detail & Related papers  (2023-07-26T08:40:46Z)
• WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognition [22.988658969594276]
  We introduce WEAR, an outdoor sports dataset for both vision- and inertial-based human activity recognition (HAR) Data from 22 participants performing a total of 18 different workout activities was collected with synchronized inertial (acceleration) and camera (egocentric video) data recorded at 11 different outside locations. Benchmark results show that through our sensor placement, each modality interestingly offers complementary strengths and weaknesses in their prediction performance.
  arXiv  Detail & Related papers  (2023-04-11T09:31:07Z)
• Multi-Channel Time-Series Person and Soft-Biometric Identification [65.83256210066787]
  This work investigates person and soft-biometrics identification from recordings of humans performing different activities using deep architectures. We evaluate the method on four datasets of multi-channel time-series human activity recognition (HAR) Soft-biometric based attribute representation shows promising results and emphasis the necessity of larger datasets.
  arXiv  Detail & Related papers  (2023-04-04T07:24:51Z)
• Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning [21.68009790164824]
  We present a multi-modal egocentric activity dataset for continual learning named UESTC-MMEA-CL. It contains synchronized data of videos, accelerometers, and gyroscopes, for 32 types of daily activities, performed by 10 participants. Results of egocentric activity recognition are reported when using separately, and jointly, three modalities: RGB, acceleration, and gyroscope.
  arXiv  Detail & Related papers  (2023-01-26T04:32:00Z)
• JRDB-Act: A Large-scale Multi-modal Dataset for Spatio-temporal Action, Social Group and Activity Detection [54.696819174421584]
  We introduce JRDB-Act, a multi-modal dataset that reflects a real distribution of human daily life actions in a university campus environment. JRDB-Act has been densely annotated with atomic actions, comprises over 2.8M action labels. JRDB-Act comes with social group identification annotations conducive to the task of grouping individuals based on their interactions in the scene.
  arXiv  Detail & Related papers  (2021-06-16T14:43:46Z)
• Sequential Weakly Labeled Multi-Activity Localization and Recognition on Wearable Sensors using Recurrent Attention Networks [13.64024154785943]
  We propose a recurrent attention network (RAN) to handle sequential weakly labeled multi-activity recognition and location tasks. Our RAN model can simultaneously infer multi-activity types from the coarse-grained sequential weak labels. It will greatly reduce the burden of manual labeling.
  arXiv  Detail & Related papers  (2020-04-13T04:57:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.