Breaking the Modality Wall: Time-step Mixup for Efficient Spiking Knowledge Transfer from Static to Event Domain

• URL: http://arxiv.org/abs/2511.12150v1
• Date: Sat, 15 Nov 2025 10:31:22 GMT
• Title: Breaking the Modality Wall: Time-step Mixup for Efficient Spiking Knowledge Transfer from Static to Event Domain
• Authors: Yuqi Xie, Shuhan Ye, Yi Yu, Chong Wang, Qixin Zhang, Jiazhen Xu, Le Shen, Yuanbin Qian, Jiangbo Qian, Guoqi Li,
• Abstract summary: Time-step Mixup Knowledge Transfer (TMKT) is a cross-modal training framework with a probabilistic Time-step Mixup (TSM) strategy.<n>TMKT exploits the asynchronous nature of SNNs by interpolating RGB and DVS inputs at various time steps to produce a smooth curriculum.<n>TMKT enables smoother knowledge transfer, helps mitigate modality mismatch during training, and achieves superior performance in spiking image classification tasks.
• Score: 28.746290149091706
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The integration of event cameras and spiking neural networks (SNNs) promises energy-efficient visual intelligence, yet scarce event data and the sparsity of DVS outputs hinder effective training. Prior knowledge transfers from RGB to DVS often underperform because the distribution gap between modalities is substantial. In this work, we present Time-step Mixup Knowledge Transfer (TMKT), a cross-modal training framework with a probabilistic Time-step Mixup (TSM) strategy. TSM exploits the asynchronous nature of SNNs by interpolating RGB and DVS inputs at various time steps to produce a smooth curriculum within each sequence, which reduces gradient variance and stabilizes optimization with theoretical analysis. To employ auxiliary supervision from TSM, TMKT introduces two lightweight modality-aware objectives, Modality Aware Guidance (MAG) for per-frame source supervision and Mixup Ratio Perception (MRP) for sequence-level mix ratio estimation, which explicitly align temporal features with the mixing schedule. TMKT enables smoother knowledge transfer, helps mitigate modality mismatch during training, and achieves superior performance in spiking image classification tasks. Extensive experiments across diverse benchmarks and multiple SNN backbones, together with ablations, demonstrate the effectiveness of our method.

Related papers

• NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective [58.345210583013454]
  Multi-Task Learning (MTL) enables a single model to learn multiple tasks simultaneously.<n> task imbalance remains a major challenge in MTL.<n>We propose a new MTL method, NTKMTL, to analyze the training dynamics in MTL.
  arXiv  Detail & Related papers  (2025-10-21T03:29:40Z)
• Time-step Mixup for Efficient Spiking Knowledge Transfer from Appearance to Event Domain [9.691720154439375]
  Time-step Mixup knowledge transfer exploits the asynchronous nature of SNNs by interpolating RGB and DVS inputs at various time-steps.<n>Our approach enables smoother knowledge transfer, alleviates modality shift during training, and achieves superior performance in spiking image classification tasks.
  arXiv  Detail & Related papers  (2025-09-16T11:02:36Z)
• Task-Aware Tuning of Time Constants in Spiking Neural Networks for Multimodal Classification [0.28272661103123253]
  Spiking Neural Networks (SNNs) are promising candidates for low-power edge computing in domains such as wearable sensing and time-series analysis.<n>Key neuronal parameter, the leaky time constant (LTC), governs temporal integration of information in Leaky Integrateand-Fire neurons.<n>This study investigates the role of LTC in a temporally adaptive feedforward SNN applied to static image, dynamic image, and biosignal time-series classification.
  arXiv  Detail & Related papers  (2025-08-23T12:18:39Z)
• WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training [64.0932926819307]
  We present Warmup-Stable and Merge (WSM), a framework that establishes a formal connection between learning rate decay and model merging.<n>WSM provides a unified theoretical foundation for emulating various decay strategies.<n>Our framework consistently outperforms the widely-adopted Warmup-Stable-Decay (WSD) approach across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-07-23T16:02:06Z)
• Towards Low-latency Event-based Visual Recognition with Hybrid Step-wise Distillation Spiking Neural Networks [50.32980443749865]
  Spiking neural networks (SNNs) have garnered significant attention for their low power consumption and high biologicalability. Current SNNs struggle to balance accuracy and latency in neuromorphic datasets. We propose Step-wise Distillation (HSD) method, tailored for neuromorphic datasets.
  arXiv  Detail & Related papers  (2024-09-19T06:52:34Z)
• ATE-SG: Alternate Through the Epochs Stochastic Gradient for Multi-Task Neural Networks [44.99833362998488]
  This paper introduces novel alternate training procedures for hard- parameter sharing Multi-Task Neural Networks (MTNNs)<n>The proposed alternate training method updates shared and task-specific weights alternately through the epochs, exploiting the multi-head architecture of the model.<n> Empirical experiments demonstrate enhanced training regularization and reduced computational demands.
  arXiv  Detail & Related papers  (2023-12-26T21:33:03Z)
• TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training [42.142924806184425]
  Mixed data samples for cross-modal contrastive learning implicitly serve as a regularizer for the contrastive loss. TiMix exhibits a comparable performance on downstream tasks, even with a reduced amount of training data and shorter training time, when benchmarked against existing methods.
  arXiv  Detail & Related papers  (2023-12-14T12:02:24Z)
• A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical Computation Offloading [62.34538208323411]
  We propose a multi-head ensemble multi-task learning (MEMTL) approach with a shared backbone and multiple prediction heads (PHs) MEMTL outperforms benchmark methods in both the inference accuracy and mean square error without requiring additional training data.
  arXiv  Detail & Related papers  (2023-09-02T11:01:16Z)
• Temporal Contrastive Learning for Spiking Neural Networks [23.963069990569714]
  Biologically inspired neural networks (SNNs) have garnered considerable attention due to their low-energy consumption and better-temporal information processing capabilities. We propose a novel method to obtain SNNs with low latency and high performance by incorporating contrastive supervision with temporal domain information.
  arXiv  Detail & Related papers  (2023-05-23T10:31:46Z)
• FedGradNorm: Personalized Federated Gradient-Normalized Multi-Task Learning [50.756991828015316]
  Multi-task learning (MTL) is a novel framework to learn several tasks simultaneously with a single shared network. We propose FedGradNorm which uses a dynamic-weighting method to normalize norms in order to balance learning speeds among different tasks.
  arXiv  Detail & Related papers  (2022-03-24T17:43:12Z)
• Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function [106.69643619725652]
  We develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results. We report state-of-the-art results for text classification task on several benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-08T21:55:22Z)
• DBS: Dynamic Batch Size For Distributed Deep Neural Network Training [19.766163856388694]
  We propose the Dynamic Batch Size (DBS) strategy for the distributedtraining of Deep Neural Networks (DNNs) Specifically, the performance of each worker is evaluatedfirst based on the fact in the previous epoch, and then the batch size and dataset partition are dynamically adjusted. The experimental results indicate that the proposed strategy can fully utilizethe performance of the cluster, reduce the training time, and have good robustness with disturbance by irrelevant tasks.
  arXiv  Detail & Related papers  (2020-07-23T07:31:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.