MobiDiary: Autoregressive Action Captioning with Wearable Devices and Wireless Signals

• URL: http://arxiv.org/abs/2601.08204v1
• Date: Tue, 13 Jan 2026 04:16:17 GMT
• Title: MobiDiary: Autoregressive Action Captioning with Wearable Devices and Wireless Signals
• Abstract summary: Human Activity Recognition (HAR) in smart homes is critical for health monitoring and assistive living.<n>We present MobiDiary, a framework that generates natural language descriptions of daily activities directly from heterogeneous physical signals.
• Score: 20.30554810750095
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human Activity Recognition (HAR) in smart homes is critical for health monitoring and assistive living. While vision-based systems are common, they face privacy concerns and environmental limitations (e.g., occlusion). In this work, we present MobiDiary, a framework that generates natural language descriptions of daily activities directly from heterogeneous physical signals (specifically IMU and Wi-Fi). Unlike conventional approaches that restrict outputs to pre-defined labels, MobiDiary produces expressive, human-readable summaries. To bridge the semantic gap between continuous, noisy physical signals and discrete linguistic descriptions, we propose a unified sensor encoder. Instead of relying on modality-specific engineering, we exploit the shared inductive biases of motion-induced signals--where both inertial and wireless data reflect underlying kinematic dynamics. Specifically, our encoder utilizes a patch-based mechanism to capture local temporal correlations and integrates heterogeneous placement embedding to unify spatial contexts across different sensors. These unified signal tokens are then fed into a Transformer-based decoder, which employs an autoregressive mechanism to generate coherent action descriptions word-by-word. We comprehensively evaluate our approach on multiple public benchmarks (XRF V2, UWash, and WiFiTAD). Experimental results demonstrate that MobiDiary effectively generalizes across modalities, achieving state-of-the-art performance on captioning metrics (e.g., BLEU@4, CIDEr, RMC) and outperforming specialized baselines in continuous action understanding.

Related papers

• The Universal Language of CSI:Unifying Wireless Sensing Across Devices and Environments [18.372157907342817]
  WiFi sensing based on Channel State Information promises ubiquitous, device-free perception.<n>Current research remains trapped in a Tower of Babel, fragmented into isolated silos.<n>We propose a foundation-model framework that treats CSI as a structured language with a learnable universal grammar.
  arXiv  Detail & Related papers  (2026-06-30T04:30:04Z)
• MIBURI: Towards Expressive Interactive Gesture Synthesis [62.45332399212876]
  Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions.<n>Existing solutions for ECAs produce rigid, low-diversity motions that are unsuitable for human-like interaction.<n>We present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue.
  arXiv  Detail & Related papers  (2026-03-03T18:59:51Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding [41.209385903282374]
  SemKey is a novel framework that enforces signal-grounded generation through four semantic objectives.<n>We move beyond standard translation metrics by adopting N-way Retrieval Accuracy and Fréchet Distance.
  arXiv  Detail & Related papers  (2026-02-09T02:47:07Z)
• Chorus: Harmonizing Context and Sensing Signals for Data-Free Model Customization in IoT [2.546351849351806]
  We propose Chorus, a data-free model customization approach that adapts models to unseen deployment conditions.<n>Chorus learns effective context representations that capture their influence on sensor data patterns.<n>Experiments show that Chorus outperforms state-of-the-art baselines by up to 11.3% in unseen contexts.
  arXiv  Detail & Related papers  (2025-12-17T08:56:21Z)
• Effectively Identifying Wi-Fi Devices through State Transitions [0.8192907805418581]
  Wi-Fi management frames reveal structured communication patterns that persist even under randomization of MAC addresses.<n>We present a novel framework for fingerprinting Wi-Fi devices based on behavioral dynamics extracted from passively observed management frames.<n>Our method achieves over 86% identification accuracy for non-randomized devices using only Wi-Fi management frames.
  arXiv  Detail & Related papers  (2025-07-03T09:35:38Z)
• ARIG: Autoregressive Interactive Head Generation for Real-time Conversations [15.886402427095515]
  Face-to-face communication, as a common human activity, motivates the research on interactive head generation.<n>Previous clip-wise generation paradigm or explicit listener/speaker generator-switching methods have limitations in future signal acquisition.<n>We propose an autoregressive (AR) based frame-wise framework called ARIG to realize the real-time generation with better interaction realism.
  arXiv  Detail & Related papers  (2025-07-01T06:38:14Z)
• DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decoding [29.643549839940025]
  We introduce DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decoding.<n>Our core idea is to frame token decoding as a conditional generation task.<n>DisCoRD achieves state-of-the-art performance, with FID of 0.032 on HumanML3D and 0.169 on KIT-ML.
  arXiv  Detail & Related papers  (2024-11-29T07:54:56Z)
• MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition [94.56755080185732]
  We propose a Motion-Aware masked autoencoder with Semantic Alignment (MASA) that integrates rich motion cues and global semantic information. Our framework can simultaneously learn local motion cues and global semantic features for comprehensive sign language representation.
  arXiv  Detail & Related papers  (2024-05-31T08:06:05Z)
• Priority-Centric Human Motion Generation in Discrete Latent Space [59.401128190423535]
  We introduce a Priority-Centric Motion Discrete Diffusion Model (M2DM) for text-to-motion generation. M2DM incorporates a global self-attention mechanism and a regularization term to counteract code collapse. We also present a motion discrete diffusion model that employs an innovative noise schedule, determined by the significance of each motion token.
  arXiv  Detail & Related papers  (2023-08-28T10:40:16Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)
• GraSens: A Gabor Residual Anti-aliasing Sensing Framework for Action Recognition using WiFi [52.530330427538885]
  WiFi-based human action recognition (HAR) has been regarded as a promising solution in applications such as smart living and remote monitoring. We propose an end-to-end Gabor residual anti-aliasing sensing network (GraSens) to directly recognize the actions using the WiFi signals from the wireless devices in diverse scenarios.
  arXiv  Detail & Related papers  (2022-05-24T10:20:16Z)
• Leveraging Semantic Scene Characteristics and Multi-Stream Convolutional Architectures in a Contextual Approach for Video-Based Visual Emotion Recognition in the Wild [31.40575057347465]
  We tackle the task of video-based visual emotion recognition in the wild. Standard methodologies that rely solely on the extraction of bodily and facial features often fall short of accurate emotion prediction. We aspire to alleviate this problem by leveraging visual context in the form of scene characteristics and attributes.
  arXiv  Detail & Related papers  (2021-05-16T17:31:59Z)
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition [131.6328804788164]
  We propose a framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), to enhance action recognition in vision-sensor modality (videos) The SAKDN uses multiple wearable-sensors as teacher modalities and uses RGB videos as student modality.
  arXiv  Detail & Related papers  (2020-09-01T03:38:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.