Energy-Efficient & Real-Time Computer Vision with Intelligent Skipping via Reconfigurable CMOS Image Sensors

• URL: http://arxiv.org/abs/2409.17341v1
• Date: Wed, 25 Sep 2024 20:32:55 GMT
• Title: Energy-Efficient & Real-Time Computer Vision with Intelligent Skipping via Reconfigurable CMOS Image Sensors
• Authors: Md Abdullah-Al Kaiser, Sreetama Sarkar, Peter A. Beerel, Akhilesh R. Jaiswal, Gourav Datta,
• Abstract summary: Video-based computer vision applications typically suffer from high energy consumption due to reading and processing all pixels in a frame, regardless of their significance. Previous works have attempted to reduce this energy by skipping input patches or pixels and using feedback from the end task to guide the skipping algorithm. This paper presents a custom-designed CMOS image sensor (CIS) system that improves energy efficiency by selectively skipping uneventful regions or rows within a frame during the sensor's readout phase.
• Score: 5.824962833043625
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current video-based computer vision (CV) applications typically suffer from high energy consumption due to reading and processing all pixels in a frame, regardless of their significance. While previous works have attempted to reduce this energy by skipping input patches or pixels and using feedback from the end task to guide the skipping algorithm, the skipping is not performed during the sensor read phase. As a result, these methods can not optimize the front-end sensor energy. Moreover, they may not be suitable for real-time applications due to the long latency of modern CV networks that are deployed in the back-end. To address this challenge, this paper presents a custom-designed reconfigurable CMOS image sensor (CIS) system that improves energy efficiency by selectively skipping uneventful regions or rows within a frame during the sensor's readout phase, and the subsequent analog-to-digital conversion (ADC) phase. A novel masking algorithm intelligently directs the skipping process in real-time, optimizing both the front-end sensor and back-end neural networks for applications including autonomous driving and augmented/virtual reality (AR/VR). Our system can also operate in standard mode without skipping, depending on application needs. We evaluate our hardware-algorithm co-design framework on object detection based on BDD100K and ImageNetVID, and gaze estimation based on OpenEDS, achieving up to 53% reduction in front-end sensor energy while maintaining state-of-the-art (SOTA) accuracy.

Related papers

• Data-Driven Pixel Control: Challenges and Prospects [13.158333009169418]
  We study a data-driven system that combines dynamic sensing at the pixel level with computer vision analytics at the video level. Our system achieves a 10X reduction in bandwidth and a 15-30X improvement in Energy-Delay Product (EDP) when activating only 30% of pixels.
  arXiv  Detail & Related papers  (2024-08-08T21:49:19Z)
• Hardware-Algorithm Re-engineering of Retinal Circuit for Intelligent Object Motion Segmentation [0.0]
  We focus on a fundamental visual feature within the mammalian retina, Object Motion Sensitivity (OMS) We present novel CMOS circuits that implement OMS functionality inside image sensors. We verify the functionality and re-configurability of the proposed CMOS circuit designs through Cadence simulations in 180nm technology.
  arXiv  Detail & Related papers  (2024-07-31T20:35:11Z)
• DemosaicFormer: Coarse-to-Fine Demosaicing Network for HybridEVS Camera [70.28702677370879]
  Hybrid Event-Based Vision Sensor (HybridEVS) is a novel sensor integrating traditional frame-based and event-based sensors. Despite its potential, the lack of Image signal processing (ISP) pipeline specifically designed for HybridEVS poses a significant challenge. We propose a coarse-to-fine framework named DemosaicFormer which comprises coarse demosaicing and pixel correction.
  arXiv  Detail & Related papers  (2024-06-12T07:20:46Z)
• Efficient and accurate neural field reconstruction using resistive memory [52.68088466453264]
  Traditional signal reconstruction methods on digital computers face both software and hardware challenges. We propose a systematic approach with software-hardware co-optimizations for signal reconstruction from sparse inputs. This work advances the AI-driven signal restoration technology and paves the way for future efficient and robust medical AI and 3D vision applications.
  arXiv  Detail & Related papers  (2024-04-15T09:33:09Z)
• Fully Quantized Always-on Face Detector Considering Mobile Image Sensors [12.806584794505751]
  Current face detectors do not fully meet the requirements for "intelligent" CMOS image sensors integrated with embedded DNNs. In this study, we aim to bridge the gap by exploring extremely low-bit lightweight face detectors.
  arXiv  Detail & Related papers  (2023-11-02T05:35:49Z)
• PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized Perception with Neural Sensors [42.18718773182277]
  Conventional image sensors digitize high-resolution images at fast frame rates, producing a large amount of data that needs to be transmitted off the sensor for further processing. We develop an efficient recurrent neural network architecture, processing PixelRNN, that encodes-temporal features on the sensor using purely binary operations. PixelRNN reduces the amount data to be transmitted off the sensor by a factor of 64x compared to conventional systems while offering competitive accuracy for hand gesture recognition and lip reading tasks.
  arXiv  Detail & Related papers  (2023-04-11T18:16:47Z)
• In-Sensor & Neuromorphic Computing are all you need for Energy Efficient Computer Vision [6.323908398583083]
  We propose an in-sensor computing hardware-software co-design framework for neuromorphic spiking neural networks (SNNs) Our approach reduces the bandwidth between sensing and processing by 12-96x and the resulting total energy by 2.32x compared to traditional CV processing.
  arXiv  Detail & Related papers  (2022-12-21T09:47:52Z)
• Braille Letter Reading: A Benchmark for Spatio-Temporal Pattern Recognition on Neuromorphic Hardware [50.380319968947035]
  Recent deep learning approaches have reached accuracy in such tasks, but their implementation on conventional embedded solutions is still computationally very and energy expensive. We propose a new benchmark for computing tactile pattern recognition at the edge through letters reading. We trained and compared feed-forward and recurrent spiking neural networks (SNNs) offline using back-propagation through time with surrogate gradients, then we deployed them on the Intel Loihimorphic chip for efficient inference. Our results show that the LSTM outperforms the recurrent SNN in terms of accuracy by 14%. However, the recurrent SNN on Loihi is 237 times more energy
  arXiv  Detail & Related papers  (2022-05-30T14:30:45Z)
• Feeling of Presence Maximization: mmWave-Enabled Virtual Reality Meets Deep Reinforcement Learning [76.46530937296066]
  This paper investigates the problem of providing ultra-reliable and energy-efficient virtual reality (VR) experiences for wireless mobile users. To ensure reliable ultra-high-definition (UHD) video frame delivery to mobile users, a coordinated multipoint (CoMP) transmission technique and millimeter wave (mmWave) communications are exploited.
  arXiv  Detail & Related papers  (2021-06-03T08:35:10Z)
• Learning Camera Miscalibration Detection [83.38916296044394]
  This paper focuses on a data-driven approach to learn the detection of miscalibration in vision sensors, specifically RGB cameras. Our contributions include a proposed miscalibration metric for RGB cameras and a novel semi-synthetic dataset generation pipeline based on this metric. By training a deep convolutional neural network, we demonstrate the effectiveness of our pipeline to identify whether a recalibration of the camera's intrinsic parameters is required or not.
  arXiv  Detail & Related papers  (2020-05-24T10:32:49Z)
• Thermal phase shifters for femtosecond laser written photonic integrated circuits [58.720142291102135]
  Photonic integrated circuits (PICs) are acknowledged as an effective solution to fulfill the demanding requirements of many practical applications in both classical and quantum optics. Phase shifters integrated in the photonic circuit offer the possibility to dynamically reconfigure its properties in order to fine tune its operation or to produce adaptive circuits. We show how thermal shifters to reconfigure photonic circuits can be solved by a careful design of the thermal shifters and by choosing the most appropriate way to drive them.
  arXiv  Detail & Related papers  (2020-04-23T20:09:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.