Efficient Flow-Guided Multi-frame De-fencing

• URL: http://arxiv.org/abs/2301.10759v1
• Date: Wed, 25 Jan 2023 18:42:59 GMT
• Title: Efficient Flow-Guided Multi-frame De-fencing
• Authors: Stavros Tsogkas, Fengjia Zhang, Allan Jepson, Alex Levinshtein
• Abstract summary: De-fencing is the algorithmic process of automatically removing such obstructions from images. We develop a framework for multi-frame de-fencing that computes high quality flow maps directly from obstructed frames.
• Score: 7.504789972841539
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Taking photographs ''in-the-wild'' is often hindered by fence obstructions that stand between the camera user and the scene of interest, and which are hard or impossible to avoid. De-fencing is the algorithmic process of automatically removing such obstructions from images, revealing the invisible parts of the scene. While this problem can be formulated as a combination of fence segmentation and image inpainting, this often leads to implausible hallucinations of the occluded regions. Existing multi-frame approaches rely on propagating information to a selected keyframe from its temporal neighbors, but they are often inefficient and struggle with alignment of severely obstructed images. In this work we draw inspiration from the video completion literature and develop a simplified framework for multi-frame de-fencing that computes high quality flow maps directly from obstructed frames and uses them to accurately align frames. Our primary focus is efficiency and practicality in a real-world setting: the input to our algorithm is a short image burst (5 frames) - a data modality commonly available in modern smartphones - and the output is a single reconstructed keyframe, with the fence removed. Our approach leverages simple yet effective CNN modules, trained on carefully generated synthetic data, and outperforms more complicated alternatives real bursts, both quantitatively and qualitatively, while running real-time.

Related papers

• Motion-Aware Video Frame Interpolation [49.49668436390514]
  We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
  arXiv  Detail & Related papers  (2024-02-05T11:00:14Z)
• Look More but Care Less in Video Recognition [57.96505328398205]
  Action recognition methods typically sample a few frames to represent each video to avoid the enormous computation. We propose Ample and Focal Network (AFNet), which is composed of two branches to utilize more frames but with less computation.
  arXiv  Detail & Related papers  (2022-11-18T02:39:56Z)
• Burst Image Restoration and Enhancement [86.08546447144377]
  The goal of Burst Image Restoration is to effectively combine complimentary cues across multiple burst frames to generate high-quality outputs. We create a set of emphpseudo-burst features that combine complimentary information from all the input burst frames to seamlessly exchange information. Our approach delivers state of the art performance on burst super-resolution and low-light image enhancement tasks.
  arXiv  Detail & Related papers  (2021-10-07T17:58:56Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)
• Motion-blurred Video Interpolation and Extrapolation [72.3254384191509]
  We present a novel framework for deblurring, interpolating and extrapolating sharp frames from a motion-blurred video in an end-to-end manner. To ensure temporal coherence across predicted frames and address potential temporal ambiguity, we propose a simple, yet effective flow-based rule.
  arXiv  Detail & Related papers  (2021-03-04T12:18:25Z)
• Multi-Stage Raw Video Denoising with Adversarial Loss and Gradient Mask [14.265454188161819]
  We propose a learning-based approach for denoising raw videos captured under low lighting conditions. We first explicitly align the neighboring frames to the current frame using a convolutional neural network (CNN) We then fuse the registered frames using another CNN to obtain the final denoised frame.
  arXiv  Detail & Related papers  (2021-03-04T06:57:48Z)
• ALANET: Adaptive Latent Attention Network forJoint Video Deblurring and Interpolation [38.52446103418748]
  We introduce a novel architecture, Adaptive Latent Attention Network (ALANET), which synthesizes sharp high frame-rate videos. We employ combination of self-attention and cross-attention module between consecutive frames in the latent space to generate optimized representation for each frame. Our method performs favorably against various state-of-the-art approaches, even though we tackle a much more difficult problem.
  arXiv  Detail & Related papers  (2020-08-31T21:11:53Z)
• All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modeling [52.425236515695914]
  State-of-the-art methods are iterative solutions interpolating one frame at the time. This work introduces a true multi-frame interpolator. It utilizes a pyramidal style network in the temporal domain to complete the multi-frame task in one-shot.
  arXiv  Detail & Related papers  (2020-07-23T02:34:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.