An Efficient Self-Supervised Framework for Long-Sequence EEG Modeling

• URL: http://arxiv.org/abs/2502.17873v2
• Date: Mon, 22 Sep 2025 19:27:18 GMT
• Title: An Efficient Self-Supervised Framework for Long-Sequence EEG Modeling
• Authors: Jiazhen Hong, Geoffrey Mackellar, Soheila Ghane,
• Abstract summary: We propose EEGM2, a self-supervised framework for EEG representation learning.<n>EEGM2 achieves state-of-the-art performance in both short- and long-sequence modeling and classification.
• Score: 2.1232375739287006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Electroencephalogram (EEG) signals generally exhibit low signal-to-noise ratio (SNR) and high inter-subject variability, making generalization across subjects and domains challenging. Recent advances in deep learning, particularly self-supervised learning with Transformer-based architectures, have shown promise in EEG representation learning. However, their quadratic computational complexity increases memory usage and slows inference, making them inefficient for modeling long-range dependencies. Moreover, most existing approaches emphasize either explicit window segmentation of the temporal signal or spectral-only input embedding while neglecting raw temporal dynamics. In this paper, we propose EEGM2, a self-supervised framework that overcomes these limitations. EEGM2 adopts a U-shaped encoder-decoder architecture integrated with Mamba-2 to achieve linear computational complexity, thereby reducing memory usage and improving inference speed. Meanwhile, the selective information propagation mechanism of Mamba-2 enables the model to effectively capture and preserve long-range dependencies in raw EEG signals, where traditional RNN or CNN architectures often struggle. Moreover, EEGM2 employs a self-supervised pre-training objective that reconstructs raw EEG using a combined L1 and spectral (Fourier-based) loss, enhancing generalization by jointly preserving temporal dynamics and spectral characteristics. Experimental results demonstrate that EEGM2 achieves state-of-the-art performance in both short- and long-sequence modeling and classification. Further evaluations show that EEGM2 consistently outperforms existing models, demonstrating strong generalization across subjects and tasks, as well as transferability across domains. Overall, EEGM2 offers an efficient and scalable solution suitable for deployment on resource-constrained brain-computer interface (BCI) devices.

Related papers

• DriveMamba: Task-Centric Scalable State Space Model for Efficient End-to-End Autonomous Driving [47.573692944838115]
  DriveMamba is a Task-Centric Scalable paradigm for efficient E2E-AD.<n>It integrates sequential task relation modeling, implicit correspondence learning and long-term temporal fusion into a single-stage Unified Mamba decoder.<n>Extensive experiments conducted on nuScenes and Bench2Drive datasets demonstrate the superiority, generalizability and great efficiency of DriveMamba.
  arXiv  Detail & Related papers  (2026-02-09T11:48:29Z)
• EUGens: Efficient, Unified, and General Dense Layers [56.498769704575544]
  We propose a new class of dense layers that generalize standard fully-connected feedforward layers, textbfEfficient, textbfUnimat and textbfGeneral dense layers (EUGens)<n>EUGens leverage random features to approximate standard FFLs and go beyond them by incorporating a direct dependence on the input norms in their computations.
  arXiv  Detail & Related papers  (2026-01-30T05:01:03Z)
• GCMCG: A Clustering-Aware Graph Attention and Expert Fusion Network for Multi-Paradigm, Multi-task, and Cross-Subject EEG Decoding [0.7871262900865523]
  Brain-Computer Interfaces (BCIs) based on Motor Imagery (MI) electroencephalogram (EEG) signals offer a direct pathway for human-machine interaction.<n>This paper proposes Graph-guided Clustering Mixture-of-Experts CNNGRUG, a novel unified framework for MI-ME EEG decoding.
  arXiv  Detail & Related papers  (2025-11-29T18:05:33Z)
• EEGReXferNet: A Lightweight Gen-AI Framework for EEG Subspace Reconstruction via Cross-Subject Transfer Learning and Channel-Aware Embedding [2.1349209400003937]
  We introduce EEGReXferNet, a lightweight framework for EEG subspace reconstruction via crosssubject-AI transfer learning.<n>EEGReXferNet employs volume conduction across neighboring channels, band-specific convolution encoding, and dynamic latent feature extraction through sliding windows.
  arXiv  Detail & Related papers  (2025-10-26T02:15:25Z)
• Bidirectional Time-Frequency Pyramid Network for Enhanced Robust EEG Classification [2.512406961007489]
  BITE (Bidirectional Time-Freq Pyramid Network) is an end-to-end unified architecture featuring robust multistream synergy, pyramid time-frequency attention (PTFA), and bidirectional adaptive convolutions.<n>As a unified architecture, it combines robust performance across both MI and SSVEP tasks with exceptional computational efficiency.<n>Our work validates that paradigm-aligned spectral-temporal processing is essential for reliable BCI systems.
  arXiv  Detail & Related papers  (2025-10-11T04:14:48Z)
• Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models [74.15250326312179]
  Diffusion Large Language Models offer efficient parallel generation and capable global modeling.<n>The dominant application ofDLLMs is hindered by the need for a statically predefined generation length.<n>We introduce DAEDAL, a novel training-free denoising strategy that enables Dynamic Adaptive Length Expansion.
  arXiv  Detail & Related papers  (2025-08-01T17:56:07Z)
• RGE-GS: Reward-Guided Expansive Driving Scene Reconstruction via Diffusion Priors [54.81109375939306]
  RGE-GS is a novel expansive reconstruction framework that synergizes diffusion-based generation with reward-guided Gaussian integration.<n>We propose a reward network that learns to identify and prioritize consistently generated patterns prior to reconstruction phases.<n>During the reconstruction process, we devise a differentiated training strategy that automatically adjust Gaussian optimization progress according to scene converge metrics.
  arXiv  Detail & Related papers  (2025-06-28T08:02:54Z)
• CodeBrain: Bridging Decoupled Tokenizer and Multi-Scale Architecture for EEG Foundation Model [33.550819280074826]
  EEG foundation models struggle with limited heterogeneous representation capacity and inefficiency in capturing multi-scale brain dependencies.<n>We propose CodeBrain, an efficient EFM structurally aligned with brain organization, trained in two stages.<n>EEGSSM combines a structured global convolution architecture and a sliding window attention mechanism to jointly model sparse long-range and local dependencies.
  arXiv  Detail & Related papers  (2025-06-10T17:20:39Z)
• Scaling Reasoning without Attention [44.42046576158219]
  We introduce ourmodel, an attention-free language model that addresses both issues through architectural and data-centric innovations.<n>Our model eliminates the need for self-attention and key-value caching, enabling fixed-memory, constant-time inference.<n>On benchmark evaluations, ourmodel-7B outperforms strong Transformer and hybrid models of comparable scale.
  arXiv  Detail & Related papers  (2025-05-28T14:52:15Z)
• ALFEE: Adaptive Large Foundation Model for EEG Representation [17.166788472910806]
  We propose the Adaptive Large Foundation model for EEG signal representation(ALFEE) framework.<n>ALFEE is a novel hybrid transformer architecture with two learning stages for robust EEG representation learning.<n>After 25,000 hours of pretraining, extensive experimental results on six downstream EEG tasks demonstrate the superior performance of ALFEE over existing models.
  arXiv  Detail & Related papers  (2025-05-07T13:32:31Z)
• CFMD: Dynamic Cross-layer Feature Fusion for Salient Object Detection [7.262250906929891]
  Cross-layer feature pyramid networks (CFPNs) have achieved notable progress in multi-scale feature fusion and boundary detail preservation for salient object detection. To address these challenges, we propose CFMD, a novel cross-layer feature pyramid network that introduces two key innovations. First, we design a context-aware feature aggregation module (CFLMA), which incorporates the state-of-the-art Mamba architecture to construct a dynamic weight distribution mechanism. Second, we introduce an adaptive dynamic upsampling unit (CFLMD) that preserves spatial details during resolution recovery.
  arXiv  Detail & Related papers  (2025-04-02T03:22:36Z)
• Paving the way for scientific foundation models: enhancing generalization and robustness in PDEs with constraint-aware pre-training [49.8035317670223]
  A scientific foundation model (SciFM) is emerging as a promising tool for learning transferable representations across diverse domains. We propose incorporating PDE residuals into pre-training either as the sole learning signal or in combination with data loss to compensate for limited or infeasible training data. Our results show that pre-training with PDE constraints significantly enhances generalization, outperforming models trained solely on solution data.
  arXiv  Detail & Related papers  (2025-03-24T19:12:39Z)
• DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving [62.62464518137153]
  DriveTransformer is a simplified E2E-AD framework for the ease of scaling up. It is composed of three unified operations: task self-attention, sensor cross-attention, temporal cross-attention. It achieves state-of-the-art performance in both simulated closed-loop benchmark Bench2Drive and real world open-loop benchmark nuScenes with high FPS.
  arXiv  Detail & Related papers  (2025-03-07T11:41:18Z)
• BHViT: Binarized Hybrid Vision Transformer [53.38894971164072]
  Model binarization has made significant progress in enabling real-time and energy-efficient computation for convolutional neural networks (CNN) We propose BHViT, a binarization-friendly hybrid ViT architecture and its full binarization model with the guidance of three important observations. Our proposed algorithm achieves SOTA performance among binary ViT methods.
  arXiv  Detail & Related papers  (2025-03-04T08:35:01Z)
• DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs [70.91804882618243]
  This paper proposes DSMoE, a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks.<n>We implement adaptive expert routing using sigmoid activation and straight-through estimators, enabling tokens to flexibly access different aspects of model knowledge.<n>Experiments on LLaMA models demonstrate that under equivalent computational constraints, DSMoE achieves superior performance compared to existing pruning and MoE approaches.
  arXiv  Detail & Related papers  (2025-02-18T02:37:26Z)
• GEFM: Graph-Enhanced EEG Foundation Model [16.335330142000657]
  Foundation models offer a promising solution by leveraging large-scale unlabeled data through pre-training.<n>We propose Graph-Enhanced EEG Foundation Model (GEFM), a novel foundation model for EEG that integrates both temporal and inter-channel information.<n>Our architecture combines Graph Neural Networks (GNNs), which effectively capture relational structures, with a masked autoencoder to enable efficient pre-training.
  arXiv  Detail & Related papers  (2024-11-29T06:57:50Z)
• BiT-MamSleep: Bidirectional Temporal Mamba for EEG Sleep Staging [9.917709200378217]
  BiT-MamSleep is a novel architecture that integrates the Triple-Resolution CNN (TRCNN) for efficient multi-scale feature extraction. BiT-MamSleep incorporates an Adaptive Feature Recalibration (AFR) module and a temporal enhancement block to dynamically refine feature importance. Experiments on four public datasets demonstrate that BiT-MamSleep significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-03T14:49:11Z)
• Efficient High-Resolution Visual Representation Learning with State Space Model for Human Pose Estimation [60.80423207808076]
  Capturing long-range dependencies while preserving high-resolution visual representations is crucial for dense prediction tasks such as human pose estimation.<n>We propose the Dynamic Visual State Space (DVSS) block, which augments visual state space models with multi-scale convolutional operations.<n>We build HRVMamba, a novel model for efficient high-resolution representation learning.
  arXiv  Detail & Related papers  (2024-10-04T06:19:29Z)
• POMONAG: Pareto-Optimal Many-Objective Neural Architecture Generator [4.09225917049674]
  Transferable NAS has emerged, generalizing the search process from dataset-dependent to task-dependent. This paper introduces POMONAG, extending DiffusionNAG via a many-optimal diffusion process. Results were validated on two search spaces -- NAS201 and MobileNetV3 -- and evaluated across 15 image classification datasets.
  arXiv  Detail & Related papers  (2024-09-30T16:05:29Z)
• EEGMamba: Bidirectional State Space Model with Mixture of Experts for EEG Multi-task Classification [1.4004287903552533]
  We introduce EEGMamba, the first universal EEG classification network to truly implement multi-task learning for EEG applications. EEGMamba seamlessly integrates the Spatio-Temporal-Adaptive (ST- adaptive) module, bidirectional Mamba, and Mixture of Experts (MoE) into a unified framework. We evaluate our model on eight publicly available EEG datasets, and the experimental results demonstrate its superior performance in four types of tasks.
  arXiv  Detail & Related papers  (2024-07-20T11:15:47Z)
• SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts [49.01990048827639]
  We introduce SEER-MoE, a framework for reducing both the memory footprint and compute requirements of pre-trained MoE models. The first stage involves pruning the total number of experts using a heavy-hitters counting guidance, while the second stage employs a regularization-based fine-tuning strategy to recover accuracy loss. Our empirical studies demonstrate the effectiveness of our method, resulting in a sparse MoEs model optimized for inference efficiency with minimal accuracy trade-offs.
  arXiv  Detail & Related papers  (2024-04-07T22:13:43Z)
• ESTformer: Transformer Utilizing Spatiotemporal Dependencies for EEG Super-resolution [14.2426667945505]
  ESTformer is an EEG framework utilizingtemporal dependencies based on the Transformer. The ESTformer applies positional encoding methods and the Multi-head Self-attention mechanism to the space and time dimensions.
  arXiv  Detail & Related papers  (2023-12-03T12:26:32Z)
• End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
  This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
  arXiv  Detail & Related papers  (2023-05-25T10:58:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.