Rethinking Random Transformers as Adaptive Sequence Smoothers for Sleep Staging

• URL: http://arxiv.org/abs/2605.09905v1
• Date: Mon, 11 May 2026 02:48:06 GMT
• Title: Rethinking Random Transformers as Adaptive Sequence Smoothers for Sleep Staging
• Abstract summary: We show that a randomly Kernel Transformer, without any training, substantially improves sleep staging performance.<n>We formalize this effect via a Random Attention Prior (RAPK), showing that random self-attention acts as an adaptive smoother.
• Score: 17.920922817772624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic sleep staging commonly adopts Transformers under the assumption that they learn complex long-range dependencies. We challenge this view by revealing a neglected property of sleep sequences: strong local temporal continuity. We show that a randomly initialized Transformer, without any training, substantially improves sleep staging performance and consistently outperforms heuristic smoothing. We formalize this effect via a Random Attention Prior Kernel (RAPK), showing that random self-attention acts as an adaptive smoother by balancing global averaging and content-based similarity while preserving stage transitions. Using two metrics, the Local Smoothness Influence Index (LSII) and the Weighted Transition Entropy (WTE), we provide evidence that most performance gains in Transformer-based sleep staging arise from architectural inductive bias rather than parameter learning. Our results suggest that sleep staging can be effectively addressed with structure-driven smoothing mechanisms rather than complex dependency modeling, enabling more efficient and edge-deployable healthcare systems for large-scale physiological monitoring.

Related papers

• When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay [6.625270757109195]
  We show that a single quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect.<n>This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes.
  arXiv  Detail & Related papers  (2026-09-08T17:46:17Z)
• SWINSleepNet: A Hierarchical Context-Aware Framework for Sleep Staging (v2) [3.911341147858216]
  Traditional epoch-level encoders fail to extract subtle temporal microstructures and intra-epoch cross-region interactions.<n>We propose SwinSleepNet, a hierarchical context-aware dual-stream framework that separately optimize intra-epoch representation learning and inter-epoch contextual modeling.<n>Our method achieves competitive overall performance, and exhibits stronger robustness and stability on difficult N1 stages and transitional epochs.
  arXiv  Detail & Related papers  (2026-08-03T13:07:49Z)
• Efficient Temporal Modeling for Mobile Sleep Staging via Lightweight Random Attention [11.02990137178079]
  We propose Random Attention (RA), a lightweight temporal modeling module based on fixed random projections.<n> RA introduces little additional parameters beyond the epoch encoder while enabling effective temporal smoothing.<n>Experiments on Sleep-EDF-20 and Sleep-EDF-78 show that RA consistently improves epoch-wise baselines by 1-3% in accuracy and F1 score.
  arXiv  Detail & Related papers  (2026-05-31T05:12:26Z)
• TEFormer: Structured Bidirectional Temporal Enhancement Modeling in Spiking Transformers [21.44931228642047]
  Spiking Neural Networks (Sss) have achieved remarkable progress, with Spiking Transformers emerging as a promising architecture for energy-efficient sequence modeling.<n>We propose TENN, the first Spiking Transformer framework that decoupling temporal modeling across its core components.
  arXiv  Detail & Related papers  (2026-01-26T08:58:36Z)
• Sleep-Based Homeostatic Regularization for Stabilizing Spike-Timing-Dependent Plasticity in Recurrent Spiking Neural Networks [14.487258585834374]
  Spike-timing-dependent plasticity (STDP) provides a biologically-plausible learning mechanism for spiking neural networks (SNNs)<n>We propose a neuromorphic regularization scheme inspired by the synaptic homeostasis hypothesis: periodic offline phases during which external inputs are suppressed, synaptic weights undergo decay toward a homeostatic baseline, and spontaneous activity enables memory consolidation.
  arXiv  Detail & Related papers  (2026-01-13T11:17:30Z)
• Parallel BiLSTM-Transformer networks for forecasting chaotic dynamics [24.960864709838436]
  This study proposes a parallel predictive framework integrating Transformer and Bidirectional Long Short-Term Memory networks.<n>The proposed hybrid model employs a dual-branch architecture, where the Transformer branch mainly captures long-range dependencies.<n>The results consistently indicate that the proposed hybrid framework outperforms both single-branch architectures across tasks.
  arXiv  Detail & Related papers  (2025-10-27T16:17:10Z)
• Powerformer: A Transformer with Weighted Causal Attention for Time-series Forecasting [50.298817606660826]
  We introduce Powerformer, a novel Transformer variant that replaces noncausal attention weights with causal weights that are reweighted according to a smooth heavy-tailed decay.<n>Our empirical results demonstrate that Powerformer achieves state-of-the-art accuracy on public time-series benchmarks.<n>Our analyses show that the model's locality bias is amplified during training, demonstrating an interplay between time-series data and power-law-based attention.
  arXiv  Detail & Related papers  (2025-02-10T04:42:11Z)
• PRformer: Pyramidal Recurrent Transformer for Multivariate Time Series Forecasting [82.03373838627606]
  Self-attention mechanism in Transformer architecture requires positional embeddings to encode temporal order in time series prediction. We argue that this reliance on positional embeddings restricts the Transformer's ability to effectively represent temporal sequences. We present a model integrating PRE with a standard Transformer encoder, demonstrating state-of-the-art performance on various real-world datasets.
  arXiv  Detail & Related papers  (2024-08-20T01:56:07Z)
• A Poisson-Gamma Dynamic Factor Model with Time-Varying Transition Dynamics [51.147876395589925]
  A non-stationary PGDS is proposed to allow the underlying transition matrices to evolve over time. A fully-conjugate and efficient Gibbs sampler is developed to perform posterior simulation. Experiments show that, in comparison with related models, the proposed non-stationary PGDS achieves improved predictive performance.
  arXiv  Detail & Related papers  (2024-02-26T04:39:01Z)
• The Lighter The Better: Rethinking Transformers in Medical Image Segmentation Through Adaptive Pruning [26.405243756778606]
  We propose to employ adaptive pruning to transformers for medical image segmentation and propose a lightweight network APFormer. To our best knowledge, this is the first work on transformer pruning for medical image analysis tasks. We prove, through ablation studies, that adaptive pruning can work as a plug-n-play module for performance improvement on other hybrid-/transformer-based methods.
  arXiv  Detail & Related papers  (2022-06-29T05:49:36Z)
• SleepTransformer: Automatic Sleep Staging with Interpretability and Uncertainty Quantification [19.51641720686276]
  Black-box skepticism is one of the main hindrances impeding deep-learning-based automatic sleep scoring from being used in clinical environments. This work proposes a sequence-to-sequence sleep-staging model, namely SleepTransformer. It is based on the transformer backbone whose self-attention scores offer interpretability of the model's decisions at both the epoch and sequence level.
  arXiv  Detail & Related papers  (2021-05-23T23:30:58Z)
• Finetuning Pretrained Transformers into RNNs [81.72974646901136]
  Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. A linear-complexity recurrent variant has proven well suited for autoregressive generation. This work aims to convert a pretrained transformer into its efficient recurrent counterpart.
  arXiv  Detail & Related papers  (2021-03-24T10:50:43Z)
• Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer [57.18185972461453]
  We introduce for the first time in surgical workflow analysis Transformer to reconsider the ignored complementary effects of spatial and temporal features for accurate phase recognition. Our framework is lightweight and processes the hybrid embeddings in parallel to achieve a high inference speed.
  arXiv  Detail & Related papers  (2021-03-17T15:12:55Z)
• Applying the Transformer to Character-level Transduction [68.91664610425114]
  The transformer has been shown to outperform recurrent neural network-based sequence-to-sequence models in various word-level NLP tasks. We show that with a large enough batch size, the transformer does indeed outperform recurrent models for character-level tasks.
  arXiv  Detail & Related papers  (2020-05-20T17:25:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.