DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models

• URL: http://arxiv.org/abs/2604.15016v1
• Date: Thu, 16 Apr 2026 13:43:51 GMT
• Title: DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models
• Abstract summary: DLink (Distilling Layer-wise and Dominant Knowledge) is a unified framework for transferring knowledge from large EEG FMs to compact students with three key innovations.<n>Experiments on four EEG benchmarks show that DLink enables students to outperform lightweight baselines while approaching fully fine-tuned FM performance at substantially lower model size and inference cost.
• Score: 45.93502533665427
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: EEG foundation models (FMs) achieve strong cross-subject and cross-task generalization but impose substantial computational and memory costs that hinder deployment on embedded BCI systems. Knowledge distillation is a natural solution; however, conventional methods fail for EEG FMs because task-relevant semantics are often distributed across intermediate layers, and aggressive dimensionality reduction can distort oscillatory structure via representational collapse and aliasing. To address these challenges, we propose DLink (Distilling Layer-wise and Dominant Knowledge), a unified framework for transferring knowledge from large EEG FMs to compact students with three key innovations: (1) a dynamic Router that adaptively aggregates teacher layers to capture dominant intermediate representations; (2) an EEG MiC student with a Mimic-then-Compress pipeline, which inherits high-dimensional teacher features and then applies structured spatio-temporal compression to avoid a heavy classification head; and (3) spectral distillation that aligns teacher-student representations in the frequency domain to regularize compression and mitigate aliasing and temporal jitter. Experiments on four EEG benchmarks show that DLink enables compact students to outperform lightweight baselines while approaching fully fine-tuned FM performance at substantially lower model size and inference cost.

Related papers

• Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Scheduling [61.28599393707943]
  Unified multimodal models jointly support understanding and generation, but incur substantial redundant computation across tokens, layers, and generation timesteps.<n>We propose a task-shared core scorer and progress-conditioned generation expansions, optimized through generation decomposition and cross-task core alignment.<n> Experiments on two representative UMM architectures demonstrate consistent quality--efficiency improvements across both tasks, retaining 98.03% of dense understanding performance with a 1.93times$ end-to-end inference speedup.
  arXiv  Detail & Related papers  (2026-08-29T14:27:55Z)
• Beyond the Aggregation Dilemma: Prior-Retaining Decoupled Learning for Multimodal Graphs [69.56407708794845]
  MultimodalAttributed Graph Learning (MAGL) integrates node attributes with structural topology via graph aggregation.<n>As pretrained encoders evolve into Large Foundation Models (LFMs), the landscape of MAGL fundamentally shifts.<n>Under high-confidence LFM priors, mandatory aggregation introduces topological intrinsic noise that overwhelms discriminative signals.<n>We propose SUPRA (Shared-Unique Prior-Retaining Architecture), a decoupled dual-pathway paradigm.<n> SUPRA processes modality-specific features through topology-agnostics while capturing structural synergy via a lightweight shared GNN, with auxiliary deep supervision counteracting gradient starvation.
  arXiv  Detail & Related papers  (2026-05-23T17:42:25Z)
• Explicit Multi-head Attention for Inter-head Interaction in Large Language Models [70.96854312026319]
  Multi-head Explicit Attention (MEA) is a simple yet effective attention variant that explicitly models cross-head interaction.<n>MEA shows strong robustness in pretraining, which allows the use of larger learning rates that lead to faster convergence.<n>This enables a practical key-value cache compression strategy that reduces KV-cache memory usage by 50% with negligible performance loss.
  arXiv  Detail & Related papers  (2026-01-27T13:45:03Z)
• GCMCG: A Clustering-Aware Graph Attention and Expert Fusion Network for Multi-Paradigm, Multi-task, and Cross-Subject EEG Decoding [0.7871262900865523]
  Brain-Computer Interfaces (BCIs) based on Motor Imagery (MI) electroencephalogram (EEG) signals offer a direct pathway for human-machine interaction.<n>This paper proposes Graph-guided Clustering Mixture-of-Experts CNNGRUG, a novel unified framework for MI-ME EEG decoding.
  arXiv  Detail & Related papers  (2025-11-29T18:05:33Z)
• UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representations [5.382357091398666]
  Unified Heterogeneous Knowledge Distillation (UHKD) is proposed as a framework that leverages intermediate features in the frequency domain for cross-architecture transfer.<n>Experiments on CIFAR-100 and ImageNet-1K demonstrate gains of 5.59% and 0.83% over the latest method.
  arXiv  Detail & Related papers  (2025-10-28T06:41:43Z)
• PHASE-Net: Physics-Grounded Harmonic Attention System for Efficient Remote Photoplethysmography Measurement [63.007237197267834]
  Existing deep learning methods are mostly physiological monitoring and lack theoretical robustness.<n>We propose a physics-informed r paradigm derived from the Navier-Stokes equations of hemodynamics, showing that the pulse signal follows a second-order system.<n>This provides a theoretical justification for using a Temporal Conal Network (TCN)<n>Phase-Net achieves state-of-the-art performance with strong efficiency, offering a theoretically grounded and deployment-ready r solution.
  arXiv  Detail & Related papers  (2025-09-29T14:36:45Z)
• DART: Dual Adaptive Refinement Transfer for Open-Vocabulary Multi-Label Recognition [59.203152078315235]
  Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify multiple seen and unseen object categories within an image.<n> Vision-Language Pre-training models offer a strong open-vocabulary foundation, but struggle with fine-grained localization under weak supervision.<n>We propose the Dual Adaptive Refinement Transfer (DART) framework to overcome these limitations.
  arXiv  Detail & Related papers  (2025-08-07T17:22:33Z)
• Large-Scale Model Enabled Semantic Communication Based on Robust Knowledge Distillation [45.347078403677216]
  Large-scale models (LSMs) can be an effective framework for semantic representation and understanding.<n>However, their direct deployment is often hindered by high computational complexity and resource requirements.<n>This paper proposes a novel knowledge distillation based semantic communication framework.
  arXiv  Detail & Related papers  (2025-08-04T07:47:18Z)
• Decoupled Competitive Framework for Semi-supervised Medical Image Segmentation [2.146676124065199]
  Semi-supervised medical image segmentation (SSMIS) is a promising solution for insufficiently annotated samples in medical domain.<n>Most approaches following the Mean Teacher (MT) or Dual Students (DS) architecture have achieved commendable results.<n>A Decoupled Competitive Framework (DCF) is elaborated in this work, which utilizes a straightforward competition mechanism for the update of EMA.<n>The DCF undergoes rigorous validation on three publicly accessible datasets, which encompass both 2D and 3D datasets.
  arXiv  Detail & Related papers  (2025-05-30T14:56:00Z)
• An Efficient Self-Supervised Framework for Long-Sequence EEG Modeling [2.1232375739287006]
  We propose EEGM2, a self-supervised framework for EEG representation learning.<n>EEGM2 achieves state-of-the-art performance in both short- and long-sequence modeling and classification.
  arXiv  Detail & Related papers  (2025-02-25T05:57:56Z)
• M3D: Manifold-based Domain Adaptation with Dynamic Distribution for Non-Deep Transfer Learning in Cross-subject and Cross-session EEG-based Emotion Recognition [11.252832459891566]
  We propose Manifold-based Domain Adaptation with Dynamic Distribution (M3D), a lightweight, non-deep transfer learning framework.<n>M3D consists of four key modules: manifold feature transformation, dynamic distribution alignment, classifier learning, and ensemble learning.<n> Experimental results show that M3D outperforms traditional non-deep learning methods with a 4.47% average improvement.
  arXiv  Detail & Related papers  (2024-04-24T03:08:25Z)
• Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching [82.71578668091914]
  This paper establishes a compact learning framework that embeds the classification and mask-refinement components into a unified deep model. We propose a novel alternative self-dual teaching (ASDT) mechanism to encourage high-quality knowledge interaction.
  arXiv  Detail & Related papers  (2021-12-17T11:56:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.