Speech Foundation Models Generalize to Time Series Tasks from Wearable Sensor Data

• URL: http://arxiv.org/abs/2509.00221v2
• Date: Mon, 20 Oct 2025 17:27:43 GMT
• Title: Speech Foundation Models Generalize to Time Series Tasks from Wearable Sensor Data
• Authors: Jaya Narain, Zakaria Aldeneh, Shirley Ren,
• Abstract summary: We show that speech foundation models learn representations that generalize beyond the speech domain.<n>We find that the convolutional feature encoders of speech models are particularly relevant for wearable sensor applications.
• Score: 6.923084335113569
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Both speech and sensor time series data encode information in both the time- and frequency- domains, like spectral powers and waveform shapelets. We show that speech foundation models learn representations that generalize beyond the speech domain and achieve state-of-the-art performance on diverse time-series tasks from wearable sensors. Probes trained on features extracted from HuBERT and wav2vec 2.0 outperform those extracted from self-supervised models trained directly on modality-specific datasets for mood classification, arrhythmia detection, and activity classification tasks. We find that the convolutional feature encoders of speech models are particularly relevant for wearable sensor applications. The proposed approach enhances performance on data-scarce time-series tasks using simple probing methods. This work takes a step toward developing generalized time-series models that unify speech and sensor modalities.

Related papers

• FusAD: Time-Frequency Fusion with Adaptive Denoising for General Time Series Analysis [92.23551599659186]
  Time series analysis plays a vital role in fields such as finance, healthcare, industry, and meteorology.<n>FusAD is a unified analysis framework designed for diverse time series tasks.
  arXiv  Detail & Related papers  (2025-12-16T04:34:27Z)
• A Unified Frequency Domain Decomposition Framework for Interpretable and Robust Time Series Forecasting [81.73338008264115]
  Current approaches for time series forecasting, whether in the time or frequency domain, predominantly use deep learning models based on linear layers or transformers.<n>We propose FIRE, a unified frequency domain decomposition framework that provides a mathematical abstraction for diverse types of time series.<n>Fire consistently outperforms state-of-the-art models on long-term forecasting benchmarks.
  arXiv  Detail & Related papers  (2025-10-11T09:59:25Z)
• FreRA: A Frequency-Refined Augmentation for Contrastive Learning on Time Series Classification [56.925103708982164]
  We present a novel perspective from the frequency domain and identify three advantages for downstream classification: global, independent, and compact.<n>We propose the lightweight yet effective Frequency Refined Augmentation (FreRA) tailored for time series contrastive learning on classification tasks.<n>FreRA consistently outperforms ten leading baselines on time series classification, anomaly detection, and transfer learning tasks.
  arXiv  Detail & Related papers  (2025-05-29T07:18:28Z)
• SensorQA: A Question Answering Benchmark for Daily-Life Monitoring [1.925154869666529]
  SensorQA is the first human-created question-answering dataset for long-term time-series sensor data for daily life monitoring.<n>We establish benchmarks for state-of-the-art AI models on this dataset and evaluate their performance on typical edge devices.<n>Our results reveal a gap between current models and optimal QA performance and efficiency, highlighting the need for new contributions.
  arXiv  Detail & Related papers  (2025-01-09T05:06:44Z)
• Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization [74.3339999119713]
  We develop a wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies.<n>Our method first scales and decomposes the input time series, then thresholds and quantizes the wavelet coefficients, and finally pre-trains an autoregressive model to forecast coefficients for the forecast horizon.
  arXiv  Detail & Related papers  (2024-12-06T18:22:59Z)
• Scaling Wearable Foundation Models [54.93979158708164]
  We investigate the scaling properties of sensor foundation models across compute, data, and model size. Using a dataset of up to 40 million hours of in-situ heart rate, heart rate variability, electrodermal activity, accelerometer, skin temperature, and altimeter per-minute data from over 165,000 people, we create LSM. Our results establish the scaling laws of LSM for tasks such as imputation, extrapolation, both across time and sensor modalities.
  arXiv  Detail & Related papers  (2024-10-17T15:08:21Z)
• SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition [13.796942110105313]
  We introduce SensorLLM, a framework that enables Large Language Models (LLMs) to perform human activity recognition (HAR) from sensor time-series data.<n> SensorLLM addresses limitations through a Sensor-Language Alignment stage, where the model aligns sensor inputs with trend descriptions.<n>In the subsequent Task-Aware Tuning stage, we refine the model for HAR classification, achieving performance that matches or surpasses state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-10-14T15:30:41Z)
• DynImp: Dynamic Imputation for Wearable Sensing Data Through Sensory and Temporal Relatedness [78.98998551326812]
  We argue that traditional methods have rarely made use of both times-series dynamics of the data as well as the relatedness of the features from different sensors. We propose a model, termed as DynImp, to handle different time point's missingness with nearest neighbors along feature axis. We show that the method can exploit the multi-modality features from related sensors and also learn from history time-series dynamics to reconstruct the data under extreme missingness.
  arXiv  Detail & Related papers  (2022-09-26T21:59:14Z)
• Voice2Series: Reprogramming Acoustic Models for Time Series Classification [65.94154001167608]
  Voice2Series is a novel end-to-end approach that reprograms acoustic models for time series classification. We show that V2S either outperforms or is tied with state-of-the-art methods on 20 tasks, and improves their average accuracy by 1.84%.
  arXiv  Detail & Related papers  (2021-06-17T07:59:15Z)
• PSEUDo: Interactive Pattern Search in Multivariate Time Series with Locality-Sensitive Hashing and Relevance Feedback [3.347485580830609]
  PSEUDo is an adaptive feature learning technique for exploring visual patterns in multi-track sequential data. Our algorithm features sub-linear training and inference time. We demonstrate superiority of PSEUDo in terms of efficiency, accuracy, and steerability.
  arXiv  Detail & Related papers  (2021-04-30T13:00:44Z)
• ESPRESSO: Entropy and ShaPe awaRe timE-Series SegmentatiOn for processing heterogeneous sensor data [5.142415132534397]
  We propose ESPRESSO, a hybrid segmentation model for multi-dimensional time-series. ESPRESSO exploits the entropy and temporal shape properties of time-series. It offers superior performance to four state-of-the-art methods across seven public datasets of wearable and wear-free sensing.
  arXiv  Detail & Related papers  (2020-07-24T10:41:20Z)
• Real Time Speech Enhancement in the Waveform Domain [99.02180506016721]
  We present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is capable of removing various kinds of background noise including stationary and non-stationary noises.
  arXiv  Detail & Related papers  (2020-06-23T09:19:13Z)
• Deep ConvLSTM with self-attention for human activity decoding using wearables [0.0]
  We propose a deep neural network architecture that captures features of multiple sensor time-series data but also selects important time points. We show the validity of the proposed approach across different data sampling strategies and demonstrate that the self-attention mechanism gave a significant improvement. The proposed methods open avenues for better decoding of human activity from multiple body sensors over extended periods time.
  arXiv  Detail & Related papers  (2020-05-02T04:30:31Z)
• Human Activity Recognition from Wearable Sensor Data Using Self-Attention [2.9023633922848586]
  We present a self-attention based neural network model for activity recognition from body-worn sensor data. We performed experiments on four popular publicly available HAR datasets: PAMAP2, Opportunity, Skoda and USC-HAD. Our model achieve significant performance improvement over recent state-of-the-art models in both benchmark test subjects and Leave-one-out-subject evaluation.
  arXiv  Detail & Related papers  (2020-03-17T14:16:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.