Real-time Hyperspectral Imaging in Hardware via Trained Metasurface Encoders

• URL: http://arxiv.org/abs/2204.02084v1
• Date: Tue, 5 Apr 2022 09:52:51 GMT
• Title: Real-time Hyperspectral Imaging in Hardware via Trained Metasurface Encoders
• Authors: Maksim Makarenko, Arturo Burguete-Lopez, Qizhou Wang, Fedor Getman, Silvio Giancola, Bernard Ghanem and Andrea Fratalocchi
• Abstract summary: Hyperspectral imaging has attracted significant attention to identify spectral signatures for image classification and automated pattern recognition in computer vision. This work introduces Hyplex, a new integrated architecture addressing the limitations discussed above. Hyplex is a CMOS-compatible, fast hyperspectral camera that replaces bulk optics with nanoscale metasurfaces inversely designed through artificial intelligence.
• Score: 55.16861072631285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperspectral imaging has attracted significant attention to identify spectral signatures for image classification and automated pattern recognition in computer vision. State-of-the-art implementations of snapshot hyperspectral imaging rely on bulky, non-integrated, and expensive optical elements, including lenses, spectrometers, and filters. These macroscopic components do not allow fast data processing for, e.g real-time and high-resolution videos. This work introduces Hyplex, a new integrated architecture addressing the limitations discussed above. Hyplex is a CMOS-compatible, fast hyperspectral camera that replaces bulk optics with nanoscale metasurfaces inversely designed through artificial intelligence. Hyplex does not require spectrometers but makes use of conventional monochrome cameras, opening up the possibility for real-time and high-resolution hyperspectral imaging at inexpensive costs. Hyplex exploits a model-driven optimization, which connects the physical metasurfaces layer with modern visual computing approaches based on end-to-end training. We design and implement a prototype version of Hyplex and compare its performance against the state-of-the-art for typical imaging tasks such as spectral reconstruction and semantic segmentation. In all benchmarks, Hyplex reports the smallest reconstruction error. We additionally present what is, to the best of our knowledge, the largest publicly available labeled hyperspectral dataset for semantic segmentation.

Related papers

• HyperDet: Generalizable Detection of Synthesized Images by Generating and Merging A Mixture of Hyper LoRAs [17.88153857572688]
  We introduce a novel and generalizable detection framework termed HyperDet. In this work, we propose a novel objective function that balances the pixel and semantic artifacts effectively. Our work paves a new way to establish generalizable domain-specific fake image detectors based on pretrained large vision models.
  arXiv  Detail & Related papers  (2024-10-08T13:43:01Z)
• Hyper-YOLO: When Visual Object Detection Meets Hypergraph Computation [74.65906322148997]
  We introduce a new object detection method that integrates hypergraph computations to capture the complex high-order correlations among visual features. Hyper-YOLO significantly outperforms the advanced YOLOv8-N and YOLOv9T with 12% $textval$ and 9% $APMoonLab improvements.
  arXiv  Detail & Related papers  (2024-08-09T01:21:15Z)
• SpectralZoom: Efficient Segmentation with an Adaptive Hyperspectral Camera [3.0175628677371935]
  We propose a vision transformer-based (ViT) algorithm that alleviates both the captured data footprint and the computational load for hyperspectral segmentation. Our camera is able to adaptively sample image regions or patches at different resolutions, instead of capturing the entire hyperspectral cube at one high resolution.
  arXiv  Detail & Related papers  (2024-06-06T17:33:23Z)
• Equivariant Imaging for Self-supervised Hyperspectral Image Inpainting [6.099927091387616]
  We introduce a novel HSI inpainting algorithm called Hyperspectral Equivariant Imaging (Hyper-EI) Results show that the proposed method achieves state-of-the-art inpainting performance compared to the existing methods.
  arXiv  Detail & Related papers  (2024-04-19T19:55:15Z)
• HerosNet: Hyperspectral Explicable Reconstruction and Optimal Sampling Deep Network for Snapshot Compressive Imaging [41.91463343106411]
  Hyperspectral imaging is an essential imaging modality for a wide range of applications, especially in remote sensing, agriculture, and medicine. Inspired by existing hyperspectral cameras that are either slow, expensive, or bulky, reconstructing hyperspectral images (HSIs) from a low-budget snapshot measurement has drawn wide attention. Recent deep unfolding networks (DUNs) for spectral snapshot sensing (SCI) have achieved remarkable success. In this paper, we propose a novel Hyperspectral Explicable Reconstruction and Optimal Sampling deep Network for SCI, dubbed HerosNet, which includes several phases under the ISTA-unfolding framework.
  arXiv  Detail & Related papers  (2021-12-12T13:42:49Z)
• Video Coding for Machine: Compact Visual Representation Compression for Intelligent Collaborative Analytics [101.35754364753409]
  Video Coding for Machines (VCM) is committed to bridging to an extent separate research tracks of video/image compression and feature compression. This paper summarizes VCM methodology and philosophy based on existing academia and industrial efforts.
  arXiv  Detail & Related papers  (2021-10-18T12:42:13Z)
• Deep Learning Approach for Hyperspectral Image Demosaicking, Spectral Correction and High-resolution RGB Reconstruction [3.0478210530038443]
  We propose a deep learning-based image demosaicking algorithm for snapshot hyperspectral images using supervised learning methods. Due to the lack of publicly available medical images, a synthetic image generation approach is proposed to simulate snapshot images from existing medical image datasets. The resulting demosaicked images are evaluated both quantitatively and qualitatively, showing clear improvements in image quality.
  arXiv  Detail & Related papers  (2021-09-03T09:50:03Z)
• Searching for Efficient Architecture for Instrument Segmentation in Robotic Surgery [58.63306322525082]
  Most applications rely on accurate real-time segmentation of high-resolution surgical images. We design a light-weight and highly-efficient deep residual architecture which is tuned to perform real-time inference of high-resolution images.
  arXiv  Detail & Related papers  (2020-07-08T21:38:29Z)
• Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery [79.69449412334188]
  In this paper, we investigate how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches. We introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data. Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images.
  arXiv  Detail & Related papers  (2020-05-18T14:25:50Z)
• Towards Coding for Human and Machine Vision: A Scalable Image Coding Approach [104.02201472370801]
  We come up with a novel image coding framework by leveraging both the compressive and the generative models. By introducing advanced generative models, we train a flexible network to reconstruct images from compact feature representations and the reference pixels. Experimental results demonstrate the superiority of our framework in both human visual quality and facial landmark detection.
  arXiv  Detail & Related papers  (2020-01-09T10:37:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.