TinyTracker: Ultra-Fast and Ultra-Low-Power Edge Vision In-Sensor for Gaze Estimation

• URL: http://arxiv.org/abs/2307.07813v5
• Date: Mon, 20 Nov 2023 08:00:38 GMT
• Title: TinyTracker: Ultra-Fast and Ultra-Low-Power Edge Vision In-Sensor for Gaze Estimation
• Authors: Pietro Bonazzi, Thomas Ruegg, Sizhen Bian, Yawei Li, Michele Magno
• Abstract summary: This work leverages one of the first "AI in sensor" vision platforms, IMX500 by Sony, to achieve ultra-fast and ultra-low-power end-to-end edge vision applications. We propose TinyTracker, a highly efficient, fully quantized model for 2D gaze estimation designed to maximize the performance of the edge vision systems considered in this study.
• Score: 11.917014372788584
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Intelligent edge vision tasks encounter the critical challenge of ensuring power and latency efficiency due to the typically heavy computational load they impose on edge platforms.This work leverages one of the first "AI in sensor" vision platforms, IMX500 by Sony, to achieve ultra-fast and ultra-low-power end-to-end edge vision applications. We evaluate the IMX500 and compare it to other edge platforms, such as the Google Coral Dev Micro and Sony Spresense, by exploring gaze estimation as a case study. We propose TinyTracker, a highly efficient, fully quantized model for 2D gaze estimation designed to maximize the performance of the edge vision systems considered in this study. TinyTracker achieves a 41x size reduction (600Kb) compared to iTracker [1] without significant loss in gaze estimation accuracy (maximum of 0.16 cm when fully quantized). TinyTracker's deployment on the Sony IMX500 vision sensor results in end-to-end latency of around 19ms. The camera takes around 17.9ms to read, process and transmit the pixels to the accelerator. The inference time of the network is 0.86ms with an additional 0.24 ms for retrieving the results from the sensor. The overall energy consumption of the end-to-end system is 4.9 mJ, including 0.06 mJ for inference. The end-to-end study shows that IMX500 is 1.7x faster than CoralMicro (19ms vs 34.4ms) and 7x more power efficient (4.9mJ VS 34.2mJ)

Related papers

• PicoSAM2: Low-Latency Segmentation In-Sensor for Edge Vision Applications [10.20223636234956]
  PicoSAM2, a lightweight (1.3M parameters, 336M MACs) promptable segmentation model optimized for edge and in-sensor execution, including the Sony IMX500.<n>On COCO and LVIS, it achieves 51.9% and 44.9% mIoU, respectively.<n>The quantized model (1.22MB) runs at 14.3 ms on the IMX500-achieving 86 MACs/cycle, making it the only model meeting both memory and compute constraints for in-sensor deployment.
  arXiv  Detail & Related papers  (2025-06-23T16:16:02Z)
• HopTrack: A Real-time Multi-Object Tracking System for Embedded Devices [11.615446679072932]
  This paper introduces HopTrack, a real-time multi-object tracking system tailored for embedded devices. Compared with the best high-end GPU modified baseline Byte (Embed), HopTrack achieves a processing speed of up to 39.29 on NVIDIA AGX Xavier.
  arXiv  Detail & Related papers  (2024-11-01T14:13:53Z)
• Q-Segment: Segmenting Images In-Sensor for Vessel-Based Medical Diagnosis [13.018482089796159]
  We present "Q-Segment", a quantized real-time segmentation algorithm, and conduct a comprehensive evaluation on a low-power edge vision platform with the Sony IMX500. Q-Segment achieves ultra-low inference time in-sensor only 0.23 ms and power consumption of only 72mW. This research contributes valuable insights into edge-based image segmentation, laying the foundation for efficient algorithms tailored to low-power environments.
  arXiv  Detail & Related papers  (2023-12-15T15:01:41Z)
• ZoomTrack: Target-aware Non-uniform Resizing for Efficient Visual Tracking [40.13014036490452]
  transformer has enabled the speed-oriented trackers to approach state-of-the-art (SOTA) performance with high-speed. We demonstrate that it is possible to narrow or even close this gap while achieving high tracking speed based on the smaller input size.
  arXiv  Detail & Related papers  (2023-10-16T05:06:13Z)
• LiteTrack: Layer Pruning with Asynchronous Feature Extraction for Lightweight and Efficient Visual Tracking [4.179339279095506]
  LiteTrack is an efficient transformer-based tracking model optimized for high-speed operations across various devices. It achieves a more favorable trade-off between accuracy and efficiency than the other lightweight trackers. LiteTrack-B9 reaches competitive 72.2% AO on GOT-10k and 82.4% AUC on TrackingNet, and operates at 171 fps on an NVIDIA 2080Ti GPU.
  arXiv  Detail & Related papers  (2023-09-17T12:01:03Z)
• InceptionNeXt: When Inception Meets ConvNeXt [167.61042926444105]
  We build a series of networks, namely IncepitonNeXt, which not only enjoy high throughputs but also maintain competitive performance. InceptionNeXt achieves 1.6x higher training throughputs than ConvNeX-T, as well as attains 0.2% top-1 accuracy improvement on ImageNet-1K.
  arXiv  Detail & Related papers  (2023-03-29T17:59:58Z)
• Ultra-low Power Deep Learning-based Monocular Relative Localization Onboard Nano-quadrotors [64.68349896377629]
  This work presents a novel autonomous end-to-end system that addresses the monocular relative localization, through deep neural networks (DNNs), of two peer nano-drones. To cope with the ultra-constrained nano-drone platform, we propose a vertically-integrated framework, including dataset augmentation, quantization, and system optimizations. Experimental results show that our DNN can precisely localize a 10cm-size target nano-drone by employing only low-resolution monochrome images, up to 2m distance.
  arXiv  Detail & Related papers  (2023-03-03T14:14:08Z)
• Efficient Visual Tracking via Hierarchical Cross-Attention Transformer [82.92565582642847]
  We present an efficient tracking method via a hierarchical cross-attention transformer named HCAT. Our model runs about 195 fps on GPU, 45 fps on CPU, and 55 fps on the edge AI platform of NVidia Jetson AGX Xavier.
  arXiv  Detail & Related papers  (2022-03-25T09:45:27Z)
• SwinTrack: A Simple and Strong Baseline for Transformer Tracking [81.65306568735335]
  We propose a fully attentional-based Transformer tracking algorithm, Swin-Transformer Tracker (SwinTrack) SwinTrack uses Transformer for both feature extraction and feature fusion, allowing full interactions between the target object and the search region for tracking. In our thorough experiments, SwinTrack sets a new record with 0.717 SUC on LaSOT, surpassing STARK by 4.6% while still running at 45 FPS.
  arXiv  Detail & Related papers  (2021-12-02T05:56:03Z)
• LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search [104.84999119090887]
  We present LightTrack, which uses neural architecture search (NAS) to design more lightweight and efficient object trackers. Comprehensive experiments show that our LightTrack is effective. It can find trackers that achieve superior performance compared to handcrafted SOTA trackers, such as SiamRPN++ and Ocean.
  arXiv  Detail & Related papers  (2021-04-29T17:55:24Z)
• Tracking Objects as Points [83.9217787335878]
  We present a simultaneous detection and tracking algorithm that is simpler, faster, and more accurate than the state of the art. Our tracker, CenterTrack, applies a detection model to a pair of images and detections from the prior frame. CenterTrack is simple, online (no peeking into the future), and real-time.
  arXiv  Detail & Related papers  (2020-04-02T17:58:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.