rETF-semiSL: Semi-Supervised Learning for Neural Collapse in Temporal Data

• URL: http://arxiv.org/abs/2508.10147v1
• Date: Wed, 13 Aug 2025 19:16:47 GMT
• Title: rETF-semiSL: Semi-Supervised Learning for Neural Collapse in Temporal Data
• Authors: Yuhan Xie, William Cappelletti, Mahsa Shoaran, Pascal Frossard,
• Abstract summary: We propose a novel semi-supervised pre-training strategy to enforce latent representations that satisfy the Neural Collapse phenomenon.<n>We show that our method significantly outperforms previous pretext tasks when applied to LSTMs, transformers, and state-space models.
• Score: 44.17657834678967
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks for time series must capture complex temporal patterns, to effectively represent dynamic data. Self- and semi-supervised learning methods show promising results in pre-training large models, which -- when finetuned for classification -- often outperform their counterparts trained from scratch. Still, the choice of pretext training tasks is often heuristic and their transferability to downstream classification is not granted, thus we propose a novel semi-supervised pre-training strategy to enforce latent representations that satisfy the Neural Collapse phenomenon observed in optimally trained neural classifiers. We use a rotational equiangular tight frame-classifier and pseudo-labeling to pre-train deep encoders with few labeled samples. Furthermore, to effectively capture temporal dynamics while enforcing embedding separability, we integrate generative pretext tasks with our method, and we define a novel sequential augmentation strategy. We show that our method significantly outperforms previous pretext tasks when applied to LSTMs, transformers, and state-space models on three multivariate time series classification datasets. These results highlight the benefit of aligning pre-training objectives with theoretically grounded embedding geometry.

Related papers

• Bridging Neural Networks and Dynamic Time Warping for Adaptive Time Series Classification [2.443957114877221]
  We develop a versatile model that adapts to cold-start conditions and becomes trainable with labeled data.<n>As a neural network, it becomes trainable when sufficient labeled data is available, while still retaining DTW's inherent interpretability.
  arXiv  Detail & Related papers  (2025-07-13T23:15:21Z)
• Large EEG-U-Transformer for Time-Step Level Detection Without Pre-Training [1.3254304182988286]
  We propose a simple U-shaped model to efficiently learn representations by capturing both local and global features.<n>Compared to other window-level classification models, our method directly outputs predictions at the time-step level.<n>Our model won 1st place in the 2025 "seizure detection challenge" organized in the International Conference on Artificial Intelligence in Epilepsy and Other Neurological Disorders.
  arXiv  Detail & Related papers  (2025-04-01T01:33:42Z)
• Self-STORM: Deep Unrolled Self-Supervised Learning for Super-Resolution Microscopy [55.2480439325792]
  We introduce deep unrolled self-supervised learning, which alleviates the need for such data by training a sequence-specific, model-based autoencoder. Our proposed method exceeds the performance of its supervised counterparts.
  arXiv  Detail & Related papers  (2024-03-25T17:40:32Z)
• Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence [56.092059713922744]
  We show that using high-level contextualized features as prediction targets can achieve superior performance. Specifically, we propose Skeleton2vec, a simple and efficient self-supervised 3D action representation learning framework. Our proposed Skeleton2vec outperforms previous methods and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2024-01-01T12:08:35Z)
• Iterative self-transfer learning: A general methodology for response time-history prediction based on small dataset [0.0]
  An iterative self-transfer learningmethod for training neural networks based on small datasets is proposed in this study. The results show that the proposed method can improve the model performance by near an order of magnitude on small datasets.
  arXiv  Detail & Related papers  (2023-06-14T18:48:04Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Brain-Inspired Spiking Neural Network for Online Unsupervised Time Series Prediction [13.521272923545409]
  We present a novel Continuous Learning-based Unsupervised Recurrent Spiking Neural Network Model (CLURSNN) CLURSNN makes online predictions by reconstructing the underlying dynamical system using Random Delay Embedding. We show that the proposed online time series prediction methodology outperforms state-of-the-art DNN models when predicting an evolving Lorenz63 dynamical system.
  arXiv  Detail & Related papers  (2023-04-10T16:18:37Z)
• PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning [109.84770951839289]
  We present PredRNN, a new recurrent network for learning visual dynamics from historical context. We show that our approach obtains highly competitive results on three standard datasets.
  arXiv  Detail & Related papers  (2021-03-17T08:28:30Z)
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning [116.24172387469994]
  We propose a higher-order LSTM model that can efficiently learn long-term correlations in the video sequence. This is accomplished through a novel tensor train module that performs prediction by combining convolutional features across time. Our results achieve state-of-the-art performance-art in a wide range of applications and datasets.
  arXiv  Detail & Related papers  (2020-02-21T05:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.