Lightweight High-Speed Photography Built on Coded Exposure and Implicit Neural Representation of Videos

• URL: http://arxiv.org/abs/2311.13134v2
• Date: Wed, 28 Aug 2024 06:15:04 GMT
• Title: Lightweight High-Speed Photography Built on Coded Exposure and Implicit Neural Representation of Videos
• Authors: Zhihong Zhang, Runzhao Yang, Jinli Suo, Yuxiao Cheng, Qionghai Dai,
• Abstract summary: The demand for compact cameras capable of recording high-speed scenes with high resolution is steadily increasing. However, achieving such capabilities often entails high bandwidth requirements, resulting in bulky, heavy systems unsuitable for low-capacity platforms. We propose a novel approach to address these challenges by combining the classical coded exposure imaging technique with the emerging implicit neural representation for videos.
• Score: 34.152901518593396
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The demand for compact cameras capable of recording high-speed scenes with high resolution is steadily increasing. However, achieving such capabilities often entails high bandwidth requirements, resulting in bulky, heavy systems unsuitable for low-capacity platforms. To address this challenge, leveraging a coded exposure setup to encode a frame sequence into a blurry snapshot and subsequently retrieve the latent sharp video presents a lightweight solution. Nevertheless, restoring motion from blur remains a formidable challenge due to the inherent ill-posedness of motion blur decomposition, the intrinsic ambiguity in motion direction, and the diverse motions present in natural videos. In this study, we propose a novel approach to address these challenges by combining the classical coded exposure imaging technique with the emerging implicit neural representation for videos. We strategically embed motion direction cues into the blurry image during the imaging process. Additionally, we develop a novel implicit neural representation based blur decomposition network to sequentially extract the latent video frames from the blurry image, leveraging the embedded motion direction cues. To validate the effectiveness and efficiency of our proposed framework, we conduct extensive experiments using benchmark datasets and real-captured blurry images. The results demonstrate that our approach significantly outperforms existing methods in terms of both quality and flexibility. The code for our work is available at .https://github.com/zhihongz/BDINR

Related papers

• Event-based Continuous Color Video Decompression from Single Frames [38.59798259847563]
  We present ContinuityCam, a novel approach to generate a continuous video from a single static RGB image, using an event camera. Our approach combines continuous long-range motion modeling with a feature-plane-based neural integration model, enabling frame prediction at arbitrary times within the events.
  arXiv  Detail & Related papers  (2023-11-30T18:59:23Z)
• Pix2HDR -- A pixel-wise acquisition and deep learning-based synthesis approach for high-speed HDR videos [2.275097126764287]
  High-speed high dynamic range () video is challenging because the camera's frame rate restricts its dynamic range. Existing methods sacrifice speed to acquire multi-exposure frames, yet misaligned motion in these frames can still pose for HDR fusion algorithms. Our method greatly enhances the vision system's adaptability and performance in dynamic conditions.
  arXiv  Detail & Related papers  (2023-10-24T19:27:35Z)
• Neural Image Re-Exposure [86.42475408644822]
  An improper shutter may lead to a blurry image, video discontinuity, or rolling shutter artifact. We propose a neural network-based image re-exposure framework. It consists of an encoder for visual latent space construction, a re-exposure module for aggregating information to neural film with a desired shutter strategy, and a decoder for 'developing' neural film into a desired image.
  arXiv  Detail & Related papers  (2023-05-23T01:55:37Z)
• Joint Video Multi-Frame Interpolation and Deblurring under Unknown Exposure Time [101.91824315554682]
  In this work, we aim ambitiously for a more realistic and challenging task - joint video multi-frame and deblurring under unknown exposure time. We first adopt a variant of supervised contrastive learning to construct an exposure-aware representation from input blurred frames. We then build our video reconstruction network upon the exposure and motion representation by progressive exposure-adaptive convolution and motion refinement.
  arXiv  Detail & Related papers  (2023-03-27T09:43:42Z)
• Towards Interpretable Video Super-Resolution via Alternating Optimization [115.85296325037565]
  We study a practical space-time video super-resolution (STVSR) problem which aims at generating a high-framerate high-resolution sharp video from a low-framerate blurry video. We propose an interpretable STVSR framework by leveraging both model-based and learning-based methods.
  arXiv  Detail & Related papers  (2022-07-21T21:34:05Z)
• Video Reconstruction from a Single Motion Blurred Image using Learned Dynamic Phase Coding [34.76550131783525]
  We propose a hybrid optical-digital method for video reconstruction using a single motion-blurred image. We use a learned dynamic phase-coding in the lens aperture during the image acquisition to encode the motion trajectories. The proposed computational camera generates a sharp frame burst of the scene at various frame rates from a single coded motion-blurred image.
  arXiv  Detail & Related papers  (2021-12-28T02:06:44Z)
• Restoration of Video Frames from a Single Blurred Image with Motion Understanding [69.90724075337194]
  We propose a novel framework to generate clean video frames from a single motion-red image. We formulate video restoration from a single blurred image as an inverse problem by setting clean image sequence and their respective motion as latent factors. Our framework is based on anblur-decoder structure with spatial transformer network modules.
  arXiv  Detail & Related papers  (2021-04-19T08:32:57Z)
• Motion-blurred Video Interpolation and Extrapolation [72.3254384191509]
  We present a novel framework for deblurring, interpolating and extrapolating sharp frames from a motion-blurred video in an end-to-end manner. To ensure temporal coherence across predicted frames and address potential temporal ambiguity, we propose a simple, yet effective flow-based rule.
  arXiv  Detail & Related papers  (2021-03-04T12:18:25Z)
• Exposure Trajectory Recovery from Motion Blur [90.75092808213371]
  Motion blur in dynamic scenes is an important yet challenging research topic. In this paper, we define exposure trajectories, which represent the motion information contained in a blurry image. A novel motion offset estimation framework is proposed to model pixel-wise displacements of the latent sharp image.
  arXiv  Detail & Related papers  (2020-10-06T05:23:33Z)
• Prior-enlightened and Motion-robust Video Deblurring [29.158836861982742]
  PRiOr-enlightened and MOTION-robust deblurring model (PROMOTION) suitable for challenging blurs. We use 3D group convolution to efficiently encode heterogeneous prior information. We also design the priors representing blur distribution, to better handle non-uniform blur-temporal domain.
  arXiv  Detail & Related papers  (2020-03-25T04:16:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.