Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval

• URL: http://arxiv.org/abs/2603.26787v1
• Date: Wed, 25 Mar 2026 08:41:07 GMT
• Title: Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval
• Abstract summary: Spiking neural networks (SNNs) have recently shown strong potential in unimodal visual and textual tasks.<n>Building a directly trained, low-energy, and high-performance SNN for multimodal applications such as image-text retrieval (ITR) remains highly challenging.<n>We present a brain-inspired Cross-Modal Spike Fusion network (CMSF) and apply it to ITR for the first time.
• Score: 79.04396446873932
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking neural networks (SNNs) have recently shown strong potential in unimodal visual and textual tasks, yet building a directly trained, low-energy, and high-performance SNN for multimodal applications such as image-text retrieval (ITR) remains highly challenging. Existing artificial neural network (ANN)-based methods often pursue richer unimodal semantics using deeper and more complex architectures, while overlooking cross-modal interaction, retrieval latency, and energy efficiency. To address these limitations, we present a brain-inspired Cross-Modal Spike Fusion network (CMSF) and apply it to ITR for the first time. The proposed spike fusion mechanism integrates unimodal features at the spike level, generating enhanced multimodal representations that act as soft supervisory signals to refine unimodal spike embeddings, effectively mitigating semantic loss within CMSF. Despite requiring only two time steps, CMSF achieves top-tier retrieval accuracy, surpassing state-of-the-art ANN counterparts while maintaining exceptionally low energy consumption and high retrieval speed. This work marks a significant step toward multimodal SNNs, offering a brain-inspired framework that unifies temporal dynamics with cross-modal alignment and provides new insights for future spiking-based multimodal research. The code is available at https://github.com/zxt6174/CMSF.

Related papers

• DBHN-Net: Dual-Branch Hybrid Neural Network For Low-Complexity Monaural Speech Enhancement [64.23650268245822]
  This work propose a DualBranch Hybrid Neural (DBHN) Network.<n>A dual-branch network integrating ANN and SNN was designed, where the SNN branch reduces power consumption while the ANN branch addresses information loss.<n>Results show that the proposed model maintains superior performance across three public datasets while achieving an average fold reduction in computational complexity.
  arXiv  Detail & Related papers  (2026-06-04T09:16:26Z)
• A Latency Coding Framework for Deep Spiking Neural Networks with Ultra-Low Latency [51.75891259753546]
  Spiking neural networks (SNNs) offer a biologically inspired computing paradigm with significant potential for energy-efficient neural processing.<n>Time-To-First-Spike (TTFS) coding encodes information through the precise timing of a neuron's first spike.<n>TTFS models lack efficient training methods, suffering from high inference latency and limited performance.<n>We present a comprehensive framework, which enables the efficient training of deep TTFS-coded SNNs by employing backpropagation throuh time (BPTT) algorithm.
  arXiv  Detail & Related papers  (2026-03-24T13:52:03Z)
• S3T-Former: A Purely Spike-Driven State-Space Topology Transformer for Skeleton Action Recognition [11.456020141268832]
  Spiking State-Space Topology Transformer (S3T-Former) is first purely spike-driven Transformer architecture specifically designed for energy-efficient skeleton action recognition.<n>To achieve true topological and temporal sparsity, we introduce Lateral Spiking Topology Routing (LSTR) for on-demand conditional spike propagation.<n>Experiments on multiple large-scale datasets demonstrate that S3T-Former achieves highly competitive accuracy while theoretically reducing energy consumption.
  arXiv  Detail & Related papers  (2026-03-18T02:09:50Z)
• Efficient Eye-based Emotion Recognition via Neural Architecture Search of Time-to-First-Spike-Coded Spiking Neural Networks [52.617096567601344]
  Time-to-first-spike (TTFS)-coded spiking neural networks (SNNs) offer a promising solution for eye-based emotion recognition.<n>TTFS-ER is the first neural architecture search framework tailored to TTFS SNNs for eye-based emotion recognition.<n>When deployed on neuromorphic hardware, TNAS-ER attains a low latency of 48 ms and an energy consumption of 0.05 J.
  arXiv  Detail & Related papers  (2025-12-02T06:35:49Z)
• SNN-Driven Multimodal Human Action Recognition via Event Camera and Skeleton Data Fusion [0.7910116766220068]
  We propose a novel Spiking Neural Network (SNN)-driven framework for multimodal human action recognition.<n>Our framework is centered on two key innovations: (1) a novel multimodal SNN architecture that employs distinct backbone networks for each modality, and (2) a pioneering SNN-based discretized information bottleneck mechanism.
  arXiv  Detail & Related papers  (2025-02-19T02:50:51Z)
• Spiking Neural Network Feature Discrimination Boosts Modality Fusion [4.888434990566422]
  We propose a feature discrimination approach for multi-modal learning with spiking neural networks (SNNs)<n>We employ deep spiking residual learning for visual modality processing and a simpler yet efficient spiking network for auditory modality processing.<n>We present our findings and evaluate our approach against similar works in the field of classification challenges.
  arXiv  Detail & Related papers  (2025-02-05T14:33:48Z)
• Multiscale fusion enhanced spiking neural network for invasive BCI neural signal decoding [13.108613110379961]
  This paper presents a novel approach utilizing a Multiscale Fusion Fusion Spiking Neural Network (MFSNN) MFSNN emulates the parallel processing and multiscale feature fusion seen in human visual perception to enable real-time, efficient, and energy-conserving neural signal decoding. MFSNN surpasses traditional artificial neural network methods, such as enhanced GRU, in both accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2024-09-14T09:53:30Z)
• SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Network [39.54624592783459]
  Spiking Neural Networks (SNNs) have emerged as a promising alternative to conventional Artificial Neural Networks (ANNs)<n>This paper presents SpikeCLIP, a novel framework designed to bridge the modality gap in spike-based computation.
  arXiv  Detail & Related papers  (2023-10-10T09:57:17Z)
• TMS: A Temporal Multi-scale Backbone Design for Speaker Embedding [60.292702363839716]
  Current SOTA backbone networks for speaker embedding are designed to aggregate multi-scale features from an utterance with multi-branch network architectures for speaker representation. We propose an effective temporal multi-scale (TMS) model where multi-scale branches could be efficiently designed in a speaker embedding network almost without increasing computational costs.
  arXiv  Detail & Related papers  (2022-03-17T05:49:35Z)
• Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision [64.71260357476602]
  Event-based vision sensors encode local pixel-wise brightness changes in streams of events rather than image frames. Recent progress in object recognition from event-based sensors has come from conversions of deep neural networks. We propose a hybrid architecture for end-to-end training of deep neural networks for event-based pattern recognition and object detection.
  arXiv  Detail & Related papers  (2021-12-06T23:45:58Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.