Neural-Network-Enhanced Metalens Camera for High-Definition, Dynamic Imaging in the Long-Wave Infrared Spectrum

• URL: http://arxiv.org/abs/2411.17139v1
• Date: Tue, 26 Nov 2024 06:09:45 GMT
• Title: Neural-Network-Enhanced Metalens Camera for High-Definition, Dynamic Imaging in the Long-Wave Infrared Spectrum
• Authors: Jing-Yang Wei, Hao Huang, Xin Zhang, De-Mao Ye, Yi Li, Le Wang, Yao-Guang Ma, Yang-Hui Li,
• Abstract summary: We develop a lightweight and cost-effective solution for the long-wave infrared imaging using a singlet. We integrate a High-Frequency-Enhancing Cycle-GAN neural network into a metalens imaging system. Our camera is capable of achieving dynamic imaging at 125 frames per second with an End Point Error value of 12.58.
• Score: 14.057686919233646
• License:
• Abstract: To provide a lightweight and cost-effective solution for the long-wave infrared imaging using a singlet, we develop a camera by integrating a High-Frequency-Enhancing Cycle-GAN neural network into a metalens imaging system. The High-Frequency-Enhancing Cycle-GAN improves the quality of the original metalens images by addressing inherent frequency loss introduced by the metalens. In addition to the bidirectional cyclic generative adversarial network, it incorporates a high-frequency adversarial learning module. This module utilizes wavelet transform to extract high-frequency components, and then establishes a high-frequency feedback loop. It enables the generator to enhance the camera outputs by integrating adversarial feedback from the high-frequency discriminator. This ensures that the generator adheres to the constraints imposed by the high-frequency adversarial loss, thereby effectively recovering the camera's frequency loss. This recovery guarantees high-fidelity image output from the camera, facilitating smooth video production. Our camera is capable of achieving dynamic imaging at 125 frames per second with an End Point Error value of 12.58. We also achieve 0.42 for Fr\'echet Inception Distance, 30.62 for Peak Signal to Noise Ratio, and 0.69 for Structural Similarity in the recorded videos.

Related papers

• Event-Enhanced Snapshot Compressive Videography at 10K FPS [33.20071708537498]
  Video snapshot compressive imaging (SCI) encodes the target dynamic scene compactly into a snapshot and reconstructs its high-speed frame sequence afterward. We propose a novel hybrid "intensity+event" imaging scheme by incorporating an event camera into a video SCI setup. We achieve high-quality videography at 0.1ms time intervals with a low-cost CMOS image sensor working at 24 FPS.
  arXiv  Detail & Related papers  (2024-04-11T08:34:10Z)
• Sign-Coded Exposure Sensing for Noise-Robust High-Speed Imaging [16.58669052286989]
  We present a novel optical compression of high-speed frames employing pixel-level sign-coded exposure. Walsh functions ensure that the noise is not amplified during high-speed frame reconstruction. Our hardware prototype demonstrated the reconstruction of 4kHz frames of a moving scene lit by ambient light only.
  arXiv  Detail & Related papers  (2023-05-05T01:03:37Z)
• Frequency Cam: Imaging Periodic Signals in Real-Time [1.774900701865444]
  We present an efficient and fully asynchronous event camera algorithm for detecting the fundamental frequency at which image pixels flicker. We discuss the important design parameters for fullsensor frequency imaging. We present Frequency Cam, an open-source implementation as a ROS node that can run on a single core of a laptop CPU at more than 50 million events per second.
  arXiv  Detail & Related papers  (2022-11-01T00:08:35Z)
• Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations [111.08941206369508]
  We propose Augmented NeRF (Aug-NeRF), which for the first time brings the power of robust data augmentations into regularizing the NeRF training. Our proposal learns to seamlessly blend worst-case perturbations into three distinct levels of the NeRF pipeline. Aug-NeRF effectively boosts NeRF performance in both novel view synthesis and underlying geometry reconstruction.
  arXiv  Detail & Related papers  (2022-07-04T02:27:07Z)
• Ev-NeRF: Event Based Neural Radiance Field [8.78321125097048]
  Ev-NeRF is a Neural Radiance Field derived from event data. We show that Ev-NeRF achieves competitive performance for intensity image reconstruction under extreme noise conditions.
  arXiv  Detail & Related papers  (2022-06-24T18:27:30Z)
• Wavelet-Based Network For High Dynamic Range Imaging [64.66969585951207]
  Existing methods, such as optical flow based and end-to-end deep learning based solutions, are error-prone either in detail restoration or ghosting artifacts removal. In this work, we propose a novel frequency-guided end-to-end deep neural network (FNet) to conduct HDR fusion in the frequency domain, and Wavelet Transform (DWT) is used to decompose inputs into different frequency bands. The low-frequency signals are used to avoid specific ghosting artifacts, while the high-frequency signals are used for preserving details.
  arXiv  Detail & Related papers  (2021-08-03T12:26:33Z)
• Focal Frequency Loss for Image Reconstruction and Synthesis [125.7135706352493]
  We show that narrowing gaps in the frequency domain can ameliorate image reconstruction and synthesis quality further. We propose a novel focal frequency loss, which allows a model to adaptively focus on frequency components that are hard to synthesize.
  arXiv  Detail & Related papers  (2020-12-23T17:32:04Z)
• Multi-level Wavelet-based Generative Adversarial Network for Perceptual Quality Enhancement of Compressed Video [51.631731922593225]
  Existing methods mainly focus on enhancing the objective quality of compressed video while ignoring its perceptual quality. We propose a novel generative adversarial network (GAN) based on multi-level wavelet packet transform (WPT) to enhance the perceptual quality of compressed video.
  arXiv  Detail & Related papers  (2020-08-02T15:01:38Z)
• Wavelet Integrated CNNs for Noise-Robust Image Classification [51.18193090255933]
  We enhance CNNs by replacing max-pooling, strided-convolution, and average-pooling with Discrete Wavelet Transform (DWT) WaveCNets, the wavelet integrated versions of VGG, ResNets, and DenseNet, achieve higher accuracy and better noise-robustness than their vanilla versions.
  arXiv  Detail & Related papers  (2020-05-07T09:10:41Z)
• Blur, Noise, and Compression Robust Generative Adversarial Networks [85.68632778835253]
  We propose blur, noise, and compression robust GAN (BNCR-GAN) to learn a clean image generator directly from degraded images. Inspired by NR-GAN, BNCR-GAN uses a multiple-generator model composed of image, blur- Kernel, noise, and quality-factor generators. We demonstrate the effectiveness of BNCR-GAN through large-scale comparative studies on CIFAR-10 and a generality analysis on FFHQ.
  arXiv  Detail & Related papers  (2020-03-17T17:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.