Toward Efficient Hyperspectral Image Processing inside Camera Pixels

• URL: http://arxiv.org/abs/2203.05696v1
• Date: Fri, 11 Mar 2022 01:06:02 GMT
• Title: Toward Efficient Hyperspectral Image Processing inside Camera Pixels
• Authors: Gourav Datta, Zihan Yin, Ajey Jacob, Akhilesh R. Jaiswal, Peter A. Beerel
• Abstract summary: Hyperspectral cameras generate a large amount of data due to the presence of hundreds of spectral bands. To mitigate this problem, we propose a form of processing-in-pixel (PIP) Our PIP-optimized custom CNN layers effectively compress the input data, significantly reducing the bandwidth required to transmit the data downstream to the HSI processing unit.
• Score: 1.6449390849183356
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperspectral cameras generate a large amount of data due to the presence of hundreds of spectral bands as opposed to only three channels (red, green, and blue) in traditional cameras. This requires a significant amount of data transmission between the hyperspectral image sensor and a processor used to classify/detect/track the images, frame by frame, expending high energy and causing bandwidth and security bottlenecks. To mitigate this problem, we propose a form of processing-in-pixel (PIP) that leverages advanced CMOS technologies to enable the pixel array to perform a wide range of complex operations required by the modern convolutional neural networks (CNN) for hyperspectral image recognition (HSI). Consequently, our PIP-optimized custom CNN layers effectively compress the input data, significantly reducing the bandwidth required to transmit the data downstream to the HSI processing unit. This reduces the average energy consumption associated with pixel array of cameras and the CNN processing unit by 25.06x and 3.90x respectively, compared to existing hardware implementations. Our custom models yield average test accuracies within 0.56% of the baseline models for the standard HSI benchmarks.

Related papers

• Data-Driven Pixel Control: Challenges and Prospects [13.158333009169418]
  We study a data-driven system that combines dynamic sensing at the pixel level with computer vision analytics at the video level. Our system achieves a 10X reduction in bandwidth and a 15-30X improvement in Energy-Delay Product (EDP) when activating only 30% of pixels.
  arXiv  Detail & Related papers  (2024-08-08T21:49:19Z)
• Spatiotemporal Attention-based Semantic Compression for Real-time Video Recognition [117.98023585449808]
  We propose a temporal attention-based autoencoder (STAE) architecture to evaluate the importance of frames and pixels in each frame. We develop a lightweight decoder that leverages a 3D-2D CNN combined to reconstruct missing information. Experimental results show that ViT_STAE can compress the video dataset H51 by 104x with only 5% accuracy loss.
  arXiv  Detail & Related papers  (2023-05-22T07:47:27Z)
• Attention-based Feature Compression for CNN Inference Offloading in Edge Computing [93.67044879636093]
  This paper studies the computational offloading of CNN inference in device-edge co-inference systems. We propose a novel autoencoder-based CNN architecture (AECNN) for effective feature extraction at end-device. Experiments show that AECNN can compress the intermediate data by more than 256x with only about 4% accuracy loss.
  arXiv  Detail & Related papers  (2022-11-24T18:10:01Z)
• Enabling ISP-less Low-Power Computer Vision [4.102254385058941]
  We release the raw version of a large-scale benchmark for generic high-level vision tasks. For ISP-less CV systems, training on raw images result in a 7.1% increase in test accuracy. We propose an energy-efficient form of analog in-pixel demosaicing that may be coupled with in-pixel CNN computations.
  arXiv  Detail & Related papers  (2022-10-11T13:47:30Z)
• Rethinking Unsupervised Neural Superpixel Segmentation [6.123324869194195]
  unsupervised learning for superpixel segmentation via CNNs has been studied. We propose three key elements to improve the efficacy of such networks. By experimenting with the BSDS500 dataset, we find evidence to the significance of our proposal.
  arXiv  Detail & Related papers  (2022-06-21T09:30:26Z)
• P2M: A Processing-in-Pixel-in-Memory Paradigm for Resource-Constrained TinyML Applications [4.102356304183255]
  High-resolution input images still need to be streamed between the camera and the AI processing unit, frame by frame, causing energy, bandwidth, and security bottlenecks. We propose a novel Processing-in-Pixel-in-memory (P2M) paradigm, that customizes the pixel array by adding support for analog multi-channel, multi-bit convolution and ReLU. Our results indicate that P2M reduces data transfer bandwidth from sensors and analog to digital conversions by 21x, and the energy-delay product (EDP) incurred in processing a MobileNetV2 model on a TinyML
  arXiv  Detail & Related papers  (2022-03-07T04:15:29Z)
• HYPER-SNN: Towards Energy-efficient Quantized Deep Spiking Neural Networks for Hyperspectral Image Classification [5.094623170336122]
  Spiking Neural Networks (SNNs) are trained with quantization-aware gradient descent to optimize weights, membrane leak, and firing thresholds. During both training and inference, the analog pixel values of a HSI are directly applied to the input layer of the SNN without the need to convert to a spike-train. We evaluate our proposal using three HSI datasets on a 3-D and a 3-D/2-D hybrid convolutional architecture.
  arXiv  Detail & Related papers  (2021-07-26T06:17:10Z)
• Learning Frequency-aware Dynamic Network for Efficient Super-Resolution [56.98668484450857]
  This paper explores a novel frequency-aware dynamic network for dividing the input into multiple parts according to its coefficients in the discrete cosine transform (DCT) domain. In practice, the high-frequency part will be processed using expensive operations and the lower-frequency part is assigned with cheap operations to relieve the computation burden. Experiments conducted on benchmark SISR models and datasets show that the frequency-aware dynamic network can be employed for various SISR neural architectures.
  arXiv  Detail & Related papers  (2021-03-15T12:54:26Z)
• CNNs for JPEGs: A Study in Computational Cost [49.97673761305336]
  Convolutional neural networks (CNNs) have achieved astonishing advances over the past decade. CNNs are capable of learning robust representations of the data directly from the RGB pixels. Deep learning methods capable of learning directly from the compressed domain have been gaining attention in recent years.
  arXiv  Detail & Related papers  (2020-12-26T15:00:10Z)
• Hyperspectral Image Super-resolution via Deep Progressive Zero-centric Residual Learning [62.52242684874278]
  Cross-modality distribution of spatial and spectral information makes the problem challenging. We propose a novel textitlightweight deep neural network-based framework, namely PZRes-Net. Our framework learns a high resolution and textitzero-centric residual image, which contains high-frequency spatial details of the scene.
  arXiv  Detail & Related papers  (2020-06-18T06:32:11Z)
• Real-Time High-Performance Semantic Image Segmentation of Urban Street Scenes [98.65457534223539]
  We propose a real-time high-performance DCNN-based method for robust semantic segmentation of urban street scenes. The proposed method achieves the accuracy of 73.6% and 68.0% mean Intersection over Union (mIoU) with the inference speed of 51.0 fps and 39.3 fps.
  arXiv  Detail & Related papers  (2020-03-11T08:45:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.