Segment, Shuffle, and Stitch: A Simple Layer for Improving Time-Series Representations

• URL: http://arxiv.org/abs/2405.20082v3
• Date: Wed, 30 Oct 2024 15:18:22 GMT
• Title: Segment, Shuffle, and Stitch: A Simple Layer for Improving Time-Series Representations
• Authors: Shivam Grover, Amin Jalali, Ali Etemad,
• Abstract summary: We propose a neural network layer called Segment, Shuffle, and Stitch (S3) to improve representation learning in time-series models. S3 works by creating non-overlapping segments from the original sequence and shuffling them in a learned manner that is optimal for the task at hand. We show that incorporating S3 results in significant improvements for the tasks of time-series classification, forecasting, and anomaly detection, improving performance on certain datasets by up to 68%.
• Score: 21.679604468538347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing approaches for learning representations of time-series keep the temporal arrangement of the time-steps intact with the presumption that the original order is the most optimal for learning. However, non-adjacent sections of real-world time-series may have strong dependencies. Accordingly, we raise the question: Is there an alternative arrangement for time-series which could enable more effective representation learning? To address this, we propose a simple plug-and-play neural network layer called Segment, Shuffle, and Stitch (S3) designed to improve representation learning in time-series models. S3 works by creating non-overlapping segments from the original sequence and shuffling them in a learned manner that is optimal for the task at hand. It then re-attaches the shuffled segments back together and performs a learned weighted sum with the original input to capture both the newly shuffled sequence along with the original sequence. S3 is modular and can be stacked to achieve different levels of granularity, and can be added to many forms of neural architectures including CNNs or Transformers with negligible computation overhead. Through extensive experiments on several datasets and state-of-the-art baselines, we show that incorporating S3 results in significant improvements for the tasks of time-series classification, forecasting, and anomaly detection, improving performance on certain datasets by up to 68\%. We also show that S3 makes the learning more stable with a smoother training loss curve and loss landscape compared to the original baseline. The code is available at https://github.com/shivam-grover/S3-TimeSeries.

Related papers

• A Stable Whitening Optimizer for Efficient Neural Network Training [101.89246340672246]
  Building on the Shampoo family of algorithms, we identify and alleviate three key issues, resulting in the proposed SPlus method.<n>First, we find that naive Shampoo is prone to divergence when matrix-inverses are cached for long periods.<n>Second, we adapt a shape-aware scaling to enable learning rate transfer across network width.<n>Third, we find that high learning rates result in large parameter noise, and propose a simple iterate-averaging scheme which unblocks faster learning.
  arXiv  Detail & Related papers  (2025-06-08T18:43:31Z)
• Learning from Streaming Video with Orthogonal Gradients [62.51504086522027]
  We address the challenge of representation learning from a continuous stream of video as input, in a self-supervised manner. This differs from the standard approaches to video learning where videos are chopped and shuffled during training in order to create a non-redundant batch. We demonstrate the drop in performance when moving from shuffled to sequential learning on three tasks.
  arXiv  Detail & Related papers  (2025-04-02T17:59:57Z)
• Time Series Representation Learning with Supervised Contrastive Temporal Transformer [8.223940676615857]
  We develop a simple, yet novel fusion model, called: textbfSupervised textbfCOntrastive textbfTemporal textbfTransformer (SCOTT) We first investigate suitable augmentation methods for various types of time series data to assist with learning change-invariant representations.
  arXiv  Detail & Related papers  (2024-03-16T03:37:19Z)
• TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling [67.02157180089573]
  Time series pre-training has recently garnered wide attention for its potential to reduce labeling expenses and benefit various downstream tasks. This paper proposes TimeSiam as a simple but effective self-supervised pre-training framework for Time series based on Siamese networks.
  arXiv  Detail & Related papers  (2024-02-04T13:10:51Z)
• Clustering based Point Cloud Representation Learning for 3D Analysis [80.88995099442374]
  We propose a clustering based supervised learning scheme for point cloud analysis. Unlike current de-facto, scene-wise training paradigm, our algorithm conducts within-class clustering on the point embedding space. Our algorithm shows notable improvements on famous point cloud segmentation datasets.
  arXiv  Detail & Related papers  (2023-07-27T03:42:12Z)
• TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders [55.00904795497786]
  We propose TimeMAE, a novel self-supervised paradigm for learning transferrable time series representations based on transformer networks. The TimeMAE learns enriched contextual representations of time series with a bidirectional encoding scheme. To solve the discrepancy issue incurred by newly injected masked embeddings, we design a decoupled autoencoder architecture.
  arXiv  Detail & Related papers  (2023-03-01T08:33:16Z)
• FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification [53.55504611255664]
  FormerTime is a hierarchical representation model for improving the classification capacity for the multivariate time series classification task. It exhibits three aspects of merits: (1) learning hierarchical multi-scale representations from time series data, (2) inheriting the strength of both transformers and convolutional networks, and (3) tacking the efficiency challenges incurred by the self-attention mechanism.
  arXiv  Detail & Related papers  (2023-02-20T07:46:14Z)
• DyG2Vec: Efficient Representation Learning for Dynamic Graphs [26.792732615703372]
  Temporal graph neural networks have shown promising results in learning inductive representations by automatically extracting temporal patterns. We present an efficient yet effective attention-based encoder that leverages temporal edge encodings and window-based subgraph sampling to generate task-agnostic embeddings.
  arXiv  Detail & Related papers  (2022-10-30T18:13:04Z)
• HyperTime: Implicit Neural Representation for Time Series [131.57172578210256]
  Implicit neural representations (INRs) have recently emerged as a powerful tool that provides an accurate and resolution-independent encoding of data. In this paper, we analyze the representation of time series using INRs, comparing different activation functions in terms of reconstruction accuracy and training convergence speed. We propose a hypernetwork architecture that leverages INRs to learn a compressed latent representation of an entire time series dataset.
  arXiv  Detail & Related papers  (2022-08-11T14:05:51Z)
• Towards Similarity-Aware Time-Series Classification [51.2400839966489]
  We study time-series classification (TSC), a fundamental task of time-series data mining. We propose Similarity-Aware Time-Series Classification (SimTSC), a framework that models similarity information with graph neural networks (GNNs)
  arXiv  Detail & Related papers  (2022-01-05T02:14:57Z)
• Curriculum Learning for Recurrent Video Object Segmentation [2.3376061255029064]
  This work explores different schedule sampling and frame skipping variations to significantly improve the performance of a recurrent architecture. Our results on the car class of the KITTI-MOTS challenge indicate that, surprisingly, an inverse schedule sampling is a better option than a classic forward one.
  arXiv  Detail & Related papers  (2020-08-15T10:51:22Z)
• Time Series Data Augmentation for Neural Networks by Time Warping with a Discriminative Teacher [17.20906062729132]
  We propose a novel time series data augmentation called guided warping. guided warping exploits the element alignment properties of Dynamic Time Warping (DTW) and shapeDTW. We evaluate the method on all 85 datasets in the 2015 UCR Time Series Archive with a deep convolutional neural network (CNN) and a recurrent neural network (RNN)
  arXiv  Detail & Related papers  (2020-04-19T06:33:44Z)
• A Deep Structural Model for Analyzing Correlated Multivariate Time Series [11.009809732645888]
  We present a deep learning structural time series model which can handle correlated multivariate time series input. The model explicitly learns/extracts the trend, seasonality, and event components. We compare our model with several state-of-the-art methods through a comprehensive set of experiments on a variety of time series data sets.
  arXiv  Detail & Related papers  (2020-01-02T18:48:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.