MR2-ByteTrack: CNN and Transformer-based Video Object Detection for AI-augmented Embedded Vision Sensor Nodes

• URL: http://arxiv.org/abs/2605.15423v1
• Date: Thu, 14 May 2026 21:13:56 GMT
• Title: MR2-ByteTrack: CNN and Transformer-based Video Object Detection for AI-augmented Embedded Vision Sensor Nodes
• Abstract summary: We introduce Multi-Resolution Rescored ByteTrack (MR2-ByteTrack), a Video Object Detection (VOD) method tailored for MCU-based embedded vision nodes.<n>MR2-ByteTrack reduces computational cost by alternating between full- and low-resolution inference.<n>Our method yields up to 55% energy savings compared to processing only full-resolution images.
• Score: 20.287412834472416
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modern smart vision sensors need on-device intelligence to process video streams, as cloud computing is often impractical due to bandwidth, latency, and privacy constraints. However, these sensory systems typically rely on ultra-low-power microcontrollers (MCUs) with limited memory and compute, making conventional video object detection methods, which require feature storage or multi-frame buffering, unfeasible. To address this challenge, we introduce Multi-Resolution Rescored ByteTrack (MR2-ByteTrack), a Video Object Detection (VOD) method tailored for MCU-based embedded vision nodes. MR2-ByteTrack reduces computational cost by alternating between full- and low-resolution inference, while linking detections across frames via ByteTrack and correcting misclassifications through the Rescore algorithm, which applies probability union rules to aggregate detection confidence scores across frames. We apply our approach to both a CNN-based detector and a Transformer-based model, demonstrating its generality across architectures with fundamentally different spatial processing. Experiments on ImageNetVID demonstrate that MR2-ByteTrack maintains accuracy, achieving mAP scores of up to 49.0 for the CNN-based models and 48.7 for the Transformer, while reducing multiply-accumulate operations by as much as 53\% for the CNNs and 32\% for the Transformer. When deployed on GAP9, an ultra-low-power RISC-V multicore MCU, our method yields up to 55\% energy savings compared to processing only full-resolution images, enabling the first real-time Transformer-based VOD on an MCU-class embedded vision node. Code available at https://github.com/Bomps4/Multi_Resolution_Rescored_ByteTrack/tree/IEEE_Access

Related papers

• Video Object Recognition in Mobile Edge Networks: Local Tracking or Edge Detection? [57.000348519630286]
  Recent advances in mobile edge computing have made it possible to offload-intensive object detection to edge servers equipped with high-accuracy neural networks.<n>This hybrid approach offers a promising solution but introduces a new challenge: deciding when to perform edge detection versus local tracking.<n>We propose the LTED-Ada in single-device setting, a deep reinforcement learning-based algorithm that adaptively selects between local tracking and edge detection.
  arXiv  Detail & Related papers  (2025-11-25T04:54:51Z)
• Rapid Salient Object Detection with Difference Convolutional Neural Networks [49.838283141381716]
  This paper addresses the challenge of deploying salient object detection (SOD) on resource-constrained devices with real-time performance.<n>We propose an efficient network design that combines traditional wisdom on SOD and the representation power of modern CNNs.
  arXiv  Detail & Related papers  (2025-07-01T20:41:05Z)
• SparseTem: Boosting the Efficiency of CNN-Based Video Encoders by Exploiting Temporal Continuity [19.900719882624028]
  We propose a memory-efficient scheduling method to eliminate memory overhead and an online adjustment mechanism to minimize accuracy degradation.<n>SparseTem achieves speedup of 1.79x for EfficientDet and 4.72x for CRNN, with minimal accuracy drop and no additional memory overhead.
  arXiv  Detail & Related papers  (2024-10-28T07:13:25Z)
• Multi-resolution Rescored ByteTrack for Video Object Detection on Ultra-low-power Embedded Systems [13.225654514930595]
  Multi-Resolution Rescored Byte-Track (MR2-ByteTrack) is a novel video object detection framework for ultra-low-power embedded processors. MR2-ByteTrack reduces the average compute load of an off-the-shelf Deep Neural Network based object detector by up to 2.25$times$. We demonstrate an average accuracy increase of 2.16% and a latency reduction of 43% on the GAP9 microcontroller.
  arXiv  Detail & Related papers  (2024-04-17T15:45:49Z)
• MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory [76.02294791513552]
  We propose a hardware-algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory. Experimental results demonstrate that our MCUFormer achieves 73.62% top-1 accuracy on ImageNet for image classification with 320KB memory.
  arXiv  Detail & Related papers  (2023-10-25T18:00:26Z)
• Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding [27.568879624013576]
  Multimodal transformer exhibits high capacity and flexibility to align image and text for visual grounding. Existing encoder-only grounding framework suffers from heavy computation due to the self-attention operation with quadratic time complexity. We present Dynamic Mutilmodal DETR (Dynamic MDETR), by decoupling the whole grounding process into encoding and decoding phases.
  arXiv  Detail & Related papers  (2022-09-28T09:43:02Z)
• TransVOD: End-to-end Video Object Detection with Spatial-Temporal Transformers [96.981282736404]
  We present TransVOD, the first end-to-end video object detection system based on spatial-temporal Transformer architectures. Our proposed TransVOD++ sets a new state-of-the-art record in terms of accuracy on ImageNet VID with 90.0% mAP. Our proposed TransVOD Lite also achieves the best speed and accuracy trade-off with 83.7% mAP while running at around 30 FPS.
  arXiv  Detail & Related papers  (2022-01-13T16:17:34Z)
• Self-supervised Video Transformer [46.295395772938214]
  From a given video, we create local and global views with varying spatial sizes and frame rates. Our self-supervised objective seeks to match the features of different views representing the same video to be intemporal. Our approach performs well on four action benchmarks and converges faster with small batch sizes.
  arXiv  Detail & Related papers  (2021-12-02T18:59:02Z)
• FQ-ViT: Fully Quantized Vision Transformer without Retraining [13.82845665713633]
  We present a systematic method to reduce the performance degradation and inference complexity of Quantized Transformers. We are the first to achieve comparable accuracy degradation (1%) on fully quantized Vision Transformers.
  arXiv  Detail & Related papers  (2021-11-27T06:20:53Z)
• Container: Context Aggregation Network [83.12004501984043]
  Recent finding shows that a simple based solution without any traditional convolutional or Transformer components can produce effective visual representations. We present the model (CONText Ion NERtwok), a general-purpose building block for multi-head context aggregation. In contrast to Transformer-based methods that do not scale well to downstream tasks that rely on larger input image resolutions, our efficient network, named modellight, can be employed in object detection and instance segmentation networks.
  arXiv  Detail & Related papers  (2021-06-02T18:09:11Z)
• Single Shot Video Object Detector [215.06904478667337]
  Single Shot Video Object Detector (SSVD) is a new architecture that novelly integrates feature aggregation into a one-stage detector for object detection in videos. For $448 times 448$ input, SSVD achieves 79.2% mAP on ImageNet VID dataset.
  arXiv  Detail & Related papers  (2020-07-07T15:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.