Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling
- URL: http://arxiv.org/abs/2402.10211v2
- Date: Tue, 16 Jul 2024 08:38:41 GMT
- Title: Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling
- Authors: Raunaq Bhirangi, Chenyu Wang, Venkatesh Pattabiraman, Carmel Majidi, Abhinav Gupta, Tess Hellebrekers, Lerrel Pinto,
- Abstract summary: We present Hierarchical State-Space Models (HiSS), a conceptually simple, new technique for continuous sequential prediction.
HiSS stacks structured state-space models on top of each other to create a temporal hierarchy.
HiSS outperforms state-of-the-art sequence models such as causal Transformers, LSTMs, S4, and Mamba by at least 23% on MSE.
- Score: 31.56197700182255
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Reasoning from sequences of raw sensory data is a ubiquitous problem across fields ranging from medical devices to robotics. These problems often involve using long sequences of raw sensor data (e.g. magnetometers, piezoresistors) to predict sequences of desirable physical quantities (e.g. force, inertial measurements). While classical approaches are powerful for locally-linear prediction problems, they often fall short when using real-world sensors. These sensors are typically non-linear, are affected by extraneous variables (e.g. vibration), and exhibit data-dependent drift. For many problems, the prediction task is exacerbated by small labeled datasets since obtaining ground-truth labels requires expensive equipment. In this work, we present Hierarchical State-Space Models (HiSS), a conceptually simple, new technique for continuous sequential prediction. HiSS stacks structured state-space models on top of each other to create a temporal hierarchy. Across six real-world sensor datasets, from tactile-based state prediction to accelerometer-based inertial measurement, HiSS outperforms state-of-the-art sequence models such as causal Transformers, LSTMs, S4, and Mamba by at least 23% on MSE. Our experiments further indicate that HiSS demonstrates efficient scaling to smaller datasets and is compatible with existing data-filtering techniques. Code, datasets and videos can be found on https://hiss-csp.github.io.
Related papers
- Synthetic location trajectory generation using categorical diffusion
models [50.809683239937584]
Diffusion models (DPMs) have rapidly evolved to be one of the predominant generative models for the simulation of synthetic data.
We propose using DPMs for the generation of synthetic individual location trajectories (ILTs) which are sequences of variables representing physical locations visited by individuals.
arXiv Detail & Related papers (2024-02-19T15:57:39Z) - Layout Sequence Prediction From Noisy Mobile Modality [53.49649231056857]
Trajectory prediction plays a vital role in understanding pedestrian movement for applications such as autonomous driving and robotics.
Current trajectory prediction models depend on long, complete, and accurately observed sequences from visual modalities.
We propose LTrajDiff, a novel approach that treats objects obstructed or out of sight as equally important as those with fully visible trajectories.
arXiv Detail & Related papers (2023-10-09T20:32:49Z) - Deep Graph Stream SVDD: Anomaly Detection in Cyber-Physical Systems [17.373668215331737]
We propose a new approach called deep graph vector data description (SVDD) for anomaly detection.
We first use a transformer to preserve both short and long temporal patterns monitoring data in temporal embeddings.
We cluster these embeddings according to sensor type and utilize them to estimate the change in connectivity between various sensors to construct a new weighted graph.
arXiv Detail & Related papers (2023-02-24T22:14:39Z) - SECOE: Alleviating Sensors Failure in Machine Learning-Coupled IoT
Systems [0.0]
This paper proposes SECOE, a proactive approach for alleviating potentially simultaneous sensor failures.
SECOE includes a novel technique to minimize the number of models in the ensemble by harnessing the correlations among sensors.
Experiments reveal that SECOE effectively preserves prediction accuracy in the presence of sensor failures.
arXiv Detail & Related papers (2022-10-05T10:58:39Z) - DynImp: Dynamic Imputation for Wearable Sensing Data Through Sensory and
Temporal Relatedness [78.98998551326812]
We argue that traditional methods have rarely made use of both times-series dynamics of the data as well as the relatedness of the features from different sensors.
We propose a model, termed as DynImp, to handle different time point's missingness with nearest neighbors along feature axis.
We show that the method can exploit the multi-modality features from related sensors and also learn from history time-series dynamics to reconstruct the data under extreme missingness.
arXiv Detail & Related papers (2022-09-26T21:59:14Z) - Energy networks for state estimation with random sensors using sparse
labels [0.0]
We propose a technique with an implicit optimization layer and a physics-based loss function that can learn from sparse labels.
Based on this technique we present two models for discrete and continuous prediction in space.
arXiv Detail & Related papers (2022-03-12T15:15:38Z) - Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals [10.866594993485226]
We propose a novel deep learning-based anomaly detection algorithm called Deep Convolutional Autoencoding Memory network (CAE-M)
We first build a Deep Convolutional Autoencoder to characterize spatial dependence of multi-sensor data with a Maximum Mean Discrepancy (MMD)
Then, we construct a Memory Network consisting of linear (Autoregressive Model) and non-linear predictions (Bigressive LSTM with Attention) to capture temporal dependence from time-series data.
arXiv Detail & Related papers (2021-07-27T06:48:20Z) - Motion Prediction Using Temporal Inception Module [96.76721173517895]
We propose a Temporal Inception Module (TIM) to encode human motion.
Our framework produces input embeddings using convolutional layers, by using different kernel sizes for different input lengths.
The experimental results on standard motion prediction benchmark datasets Human3.6M and CMU motion capture dataset show that our approach consistently outperforms the state of the art methods.
arXiv Detail & Related papers (2020-10-06T20:26:01Z) - Superiority of Simplicity: A Lightweight Model for Network Device
Workload Prediction [58.98112070128482]
We propose a lightweight solution for series prediction based on historic observations.
It consists of a heterogeneous ensemble method composed of two models - a neural network and a mean predictor.
It achieves an overall $R2$ score of 0.10 on the available FedCSIS 2020 challenge dataset.
arXiv Detail & Related papers (2020-07-07T15:44:16Z) - ST-MNIST -- The Spiking Tactile MNIST Neuromorphic Dataset [13.270250399169104]
We debut a novel neuromorphic Spiking Tactile MNIST dataset, which comprises handwritten digits obtained by human participants writing on a tactile neuromorphic sensor array.
We also describe an initial effort to evaluate our ST-MNIST dataset using existing artificial spiking and neural network models.
arXiv Detail & Related papers (2020-05-08T23:44:14Z) - Deep Soft Procrustes for Markerless Volumetric Sensor Alignment [81.13055566952221]
In this work, we improve markerless data-driven correspondence estimation to achieve more robust multi-sensor spatial alignment.
We incorporate geometric constraints in an end-to-end manner into a typical segmentation based model and bridge the intermediate dense classification task with the targeted pose estimation one.
Our model is experimentally shown to achieve similar results with marker-based methods and outperform the markerless ones, while also being robust to the pose variations of the calibration structure.
arXiv Detail & Related papers (2020-03-23T10:51:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.