Digital Gimbal: End-to-end Deep Image Stabilization with Learnable Exposure Times

• URL: http://arxiv.org/abs/2012.04515v3
• Date: Mon, 10 May 2021 12:31:23 GMT
• Title: Digital Gimbal: End-to-end Deep Image Stabilization with Learnable Exposure Times
• Authors: Omer Dahary, Matan Jacoby, Alex M. Bronstein
• Abstract summary: We digitally emulate a mechanically stabilized system from the input of a fast unstabilized camera. To exploit the trade-off between motion blur at long exposures and low SNR at short exposures, we train a CNN that estimates a sharp high-SNR image.
• Score: 2.6396287656676733
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mechanical image stabilization using actuated gimbals enables capturing long-exposure shots without suffering from blur due to camera motion. These devices, however, are often physically cumbersome and expensive, limiting their widespread use. In this work, we propose to digitally emulate a mechanically stabilized system from the input of a fast unstabilized camera. To exploit the trade-off between motion blur at long exposures and low SNR at short exposures, we train a CNN that estimates a sharp high-SNR image by aggregating a burst of noisy short-exposure frames, related by unknown motion. We further suggest learning the burst's exposure times in an end-to-end manner, thus balancing the noise and blur across the frames. We demonstrate this method's advantage over the traditional approach of deblurring a single image or denoising a fixed-exposure burst on both synthetic and real data.

Related papers

• Non-Uniform Exposure Imaging via Neuromorphic Shutter Control [9.519068512865463]
  We propose a novel Neuromorphic Shutter Control (NSC) system to avoid motion blurs and alleviate instant noises. To stabilize the inconsistent Signal-to-Noise Ratio (SNR), we propose an event-based image denoising network within a self-supervised learning paradigm.
  arXiv  Detail & Related papers  (2024-04-22T08:28:41Z)
• Motion Blur Decomposition with Cross-shutter Guidance [33.72961622720793]
  Motion blur is an artifact under insufficient illumination where exposure time has to be prolonged so as to collect more photons for a bright enough image. Recent researches have aimed at decomposing a blurry image into multiple sharp images with spatial and temporal coherence. We propose to utilize the ordered scanline-wise delay in a rolling shutter image to robustify motion decomposition of a single blurry image.
  arXiv  Detail & Related papers  (2024-04-01T13:55:40Z)
• Dual-Camera Joint Deblurring-Denoising [24.129908866882346]
  We propose a novel dual-camera method for obtaining a high-quality image. Our method uses a synchronized burst of short exposure images captured by one camera and a long exposure image simultaneously captured by another. Our method is able to achieve state-of-the-art results on synthetic dual-camera images from the GoPro dataset with five times fewer training parameters compared to the next best method.
  arXiv  Detail & Related papers  (2023-09-16T00:58:40Z)
• Recovering Continuous Scene Dynamics from A Single Blurry Image with Events [58.7185835546638]
  An Implicit Video Function (IVF) is learned to represent a single motion blurred image with concurrent events. A dual attention transformer is proposed to efficiently leverage merits from both modalities. The proposed network is trained only with the supervision of ground-truth images of limited referenced timestamps.
  arXiv  Detail & Related papers  (2023-04-05T18:44:17Z)
• Joint Video Multi-Frame Interpolation and Deblurring under Unknown Exposure Time [101.91824315554682]
  In this work, we aim ambitiously for a more realistic and challenging task - joint video multi-frame and deblurring under unknown exposure time. We first adopt a variant of supervised contrastive learning to construct an exposure-aware representation from input blurred frames. We then build our video reconstruction network upon the exposure and motion representation by progressive exposure-adaptive convolution and motion refinement.
  arXiv  Detail & Related papers  (2023-03-27T09:43:42Z)
• Learn to See Faster: Pushing the Limits of High-Speed Camera with Deep Underexposed Image Denoising [12.507566152678857]
  The ability to record high-fidelity videos at high acquisition rates is central to the study of fast moving phenomena. The difficulty of imaging fast moving scenes lies in a trade-off between motion blur and underexposure noise. We propose to address this trade-off by treating the problem of high-speed imaging as an underexposed image denoising problem.
  arXiv  Detail & Related papers  (2022-11-29T09:10:50Z)
• Self-Supervised Image Restoration with Blurry and Noisy Pairs [66.33313180767428]
  Images with high ISO usually have inescapable noise, while the long-exposure ones may be blurry due to camera shake or object motion. Existing solutions generally suggest to seek a balance between noise and blur, and learn denoising or deblurring models under either full- or self-supervision. We propose jointly leveraging the short-exposure noisy image and the long-exposure blurry image for better image restoration.
  arXiv  Detail & Related papers  (2022-11-14T12:57:41Z)
• Robust Scene Inference under Noise-Blur Dual Corruptions [20.0721386176278]
  Scene inference under low-light is a challenging problem due to severe noise in the captured images. With the rise of cameras capable of capturing multiple exposures of the same scene simultaneously, it is possible to overcome this trade-off. We propose a method to leverage these multi exposure captures for robust inference under low-light and motion.
  arXiv  Detail & Related papers  (2022-07-24T02:52:00Z)
• Learning Spatially Varying Pixel Exposures for Motion Deblurring [49.07867902677453]
  We present a novel approach of leveraging spatially varying pixel exposures for motion deblurring. Our work illustrates the promising role that focal-plane sensor--processors can play in the future of computational imaging.
  arXiv  Detail & Related papers  (2022-04-14T23:41:49Z)
• Neural Re-rendering for Full-frame Video Stabilization [144.9918806873405]
  We present an algorithm for full-frame video stabilization by first estimating dense warp fields. Full-frame stabilized frames can then be synthesized by fusing warped contents from neighboring frames.
  arXiv  Detail & Related papers  (2021-02-11T18:59:45Z)
• Exposure Trajectory Recovery from Motion Blur [90.75092808213371]
  Motion blur in dynamic scenes is an important yet challenging research topic. In this paper, we define exposure trajectories, which represent the motion information contained in a blurry image. A novel motion offset estimation framework is proposed to model pixel-wise displacements of the latent sharp image.
  arXiv  Detail & Related papers  (2020-10-06T05:23:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.