SMTrack: End-to-End Trained Spiking Neural Networks for Multi-Object Tracking in RGB Videos

• URL: http://arxiv.org/abs/2508.14607v1
• Date: Wed, 20 Aug 2025 10:47:37 GMT
• Title: SMTrack: End-to-End Trained Spiking Neural Networks for Multi-Object Tracking in RGB Videos
• Authors: Pengzhi Zhong, Xinzhe Wang, Dan Zeng, Qihua Zhou, Feixiang He, Shuiwang Li,
• Abstract summary: Brain-inspired Spiking Neural Networks (SNNs) exhibit significant potential for low-power computation.<n>Their application in visual tasks remains largely confined to image classification, object detection, and event-based tracking.<n>We propose SMTrack-the first directly trained deep SNN framework for end-to-end multi-object tracking on standard RGB videos.
• Score: 8.673924616309698
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Brain-inspired Spiking Neural Networks (SNNs) exhibit significant potential for low-power computation, yet their application in visual tasks remains largely confined to image classification, object detection, and event-based tracking. In contrast, real-world vision systems still widely use conventional RGB video streams, where the potential of directly-trained SNNs for complex temporal tasks such as multi-object tracking (MOT) remains underexplored. To address this challenge, we propose SMTrack-the first directly trained deep SNN framework for end-to-end multi-object tracking on standard RGB videos. SMTrack introduces an adaptive and scale-aware Normalized Wasserstein Distance loss (Asa-NWDLoss) to improve detection and localization performance under varying object scales and densities. Specifically, the method computes the average object size within each training batch and dynamically adjusts the normalization factor, thereby enhancing sensitivity to small objects. For the association stage, we incorporate the TrackTrack identity module to maintain robust and consistent object trajectories. Extensive evaluations on BEE24, MOT17, MOT20, and DanceTrack show that SMTrack achieves performance on par with leading ANN-based MOT methods, advancing robust and accurate SNN-based tracking in complex scenarios.

Related papers

• Exploring Dynamic Transformer for Efficient Object Tracking [58.120191254379854]
  We propose DyTrack, a dynamic transformer framework for efficient tracking.<n>DyTrack automatically learns to configure proper reasoning routes for various inputs, gaining better utilization of the available computational budget.<n>Experiments on multiple benchmarks demonstrate that DyTrack achieves promising speed-precision trade-offs with only a single model.
  arXiv  Detail & Related papers  (2024-03-26T12:31:58Z)
• SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Features [52.213656737672935]
  SpikeMOT is an event-based multi-object tracker. SpikeMOT uses spiking neural networks to extract sparsetemporal features from event streams associated with objects.
  arXiv  Detail & Related papers  (2023-09-29T05:13:43Z)
• Unifying Tracking and Image-Video Object Detection [54.91658924277527]
  TrIVD (Tracking and Image-Video Detection) is the first framework that unifies image OD, video OD, and MOT within one end-to-end model. To handle the discrepancies and semantic overlaps of category labels, TrIVD formulates detection/tracking as grounding and reasons about object categories.
  arXiv  Detail & Related papers  (2022-11-20T20:30:28Z)
• RLM-Tracking: Online Multi-Pedestrian Tracking Supported by Relative Location Mapping [5.9669075749248774]
  Problem of multi-object tracking is a fundamental computer vision research focus, widely used in public safety, transport, autonomous vehicles, robotics, and other regions involving artificial intelligence. In this paper, we design a new multi-object tracker for the above issues that contains an object textbfRelative Location Mapping (RLM) model and textbfTarget Region Density (TRD) model. The new tracker is more sensitive to the differences in position relationships between objects. It can introduce low-score detection frames into different regions in real-time according to the density of object
  arXiv  Detail & Related papers  (2022-10-19T11:37:14Z)
• Unified Transformer Tracker for Object Tracking [58.65901124158068]
  We present the Unified Transformer Tracker (UTT) to address tracking problems in different scenarios with one paradigm. A track transformer is developed in our UTT to track the target in both Single Object Tracking (SOT) and Multiple Object Tracking (MOT)
  arXiv  Detail & Related papers  (2022-03-29T01:38:49Z)
• TDIOT: Target-driven Inference for Deep Video Object Tracking [0.2457872341625575]
  In this work, we adopt the pre-trained Mask R-CNN deep object detector as the baseline. We introduce a novel inference architecture placed on top of FPN-ResNet101 backbone of Mask R-CNN to jointly perform detection and tracking. The proposed single object tracker, TDIOT, applies an appearance similarity-based temporal matching for data association.
  arXiv  Detail & Related papers  (2021-03-19T20:45:06Z)
• TRAT: Tracking by Attention Using Spatio-Temporal Features [14.520067060603209]
  We propose a two-stream deep neural network tracker that uses both spatial and temporal features. Our architecture is developed over ATOM tracker and contains two backbones: (i) 2D-CNN network to capture appearance features and (ii) 3D-CNN network to capture motion features.
  arXiv  Detail & Related papers  (2020-11-18T20:11:12Z)
• Object Tracking through Residual and Dense LSTMs [67.98948222599849]
  Deep learning-based trackers based on LSTMs (Long Short-Term Memory) recurrent neural networks have emerged as a powerful alternative. DenseLSTMs outperform Residual and regular LSTM, and offer a higher resilience to nuisances. Our case study supports the adoption of residual-based RNNs for enhancing the robustness of other trackers.
  arXiv  Detail & Related papers  (2020-06-22T08:20:17Z)
• Robust Visual Object Tracking with Two-Stream Residual Convolutional Networks [62.836429958476735]
  We propose a Two-Stream Residual Convolutional Network (TS-RCN) for visual tracking. Our TS-RCN can be integrated with existing deep learning based visual trackers. To further improve the tracking performance, we adopt a "wider" residual network ResNeXt as its feature extraction backbone.
  arXiv  Detail & Related papers  (2020-05-13T19:05:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.