Burstormer: Burst Image Restoration and Enhancement Transformer

• URL: http://arxiv.org/abs/2304.01194v1
• Date: Mon, 3 Apr 2023 17:58:44 GMT
• Title: Burstormer: Burst Image Restoration and Enhancement Transformer
• Authors: Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Shahbaz Khan, Ming-Hsuan Yang
• Abstract summary: On a shutter press, modern handheld cameras capture multiple images in rapid succession and merge them to generate a single image. The challenge is to properly align the successive image shots and merge their complimentary information to achieve high-quality outputs. We propose Burstormer: a novel transformer-based architecture for burst image restoration and enhancement.
• Score: 117.56199661345993
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On a shutter press, modern handheld cameras capture multiple images in rapid succession and merge them to generate a single image. However, individual frames in a burst are misaligned due to inevitable motions and contain multiple degradations. The challenge is to properly align the successive image shots and merge their complimentary information to achieve high-quality outputs. Towards this direction, we propose Burstormer: a novel transformer-based architecture for burst image restoration and enhancement. In comparison to existing works, our approach exploits multi-scale local and non-local features to achieve improved alignment and feature fusion. Our key idea is to enable inter-frame communication in the burst neighborhoods for information aggregation and progressive fusion while modeling the burst-wide context. However, the input burst frames need to be properly aligned before fusing their information. Therefore, we propose an enhanced deformable alignment module for aligning burst features with regards to the reference frame. Unlike existing methods, the proposed alignment module not only aligns burst features but also exchanges feature information and maintains focused communication with the reference frame through the proposed reference-based feature enrichment mechanism, which facilitates handling complex motions. After multi-level alignment and enrichment, we re-emphasize on inter-frame communication within burst using a cyclic burst sampling module. Finally, the inter-frame information is aggregated using the proposed burst feature fusion module followed by progressive upsampling. Our Burstormer outperforms state-of-the-art methods on burst super-resolution, burst denoising and burst low-light enhancement. Our codes and pretrained models are available at https:// github.com/akshaydudhane16/Burstormer

Related papers

• Aggregating Long-term Sharp Features via Hybrid Transformers for Video Deblurring [76.54162653678871]
  We propose a video deblurring method that leverages both neighboring frames and present sharp frames using hybrid Transformers for feature aggregation. Our proposed method outperforms state-of-the-art video deblurring methods as well as event-driven video deblurring methods in terms of quantitative metrics and visual quality.
  arXiv  Detail & Related papers  (2023-09-13T16:12:11Z)
• Gated Multi-Resolution Transfer Network for Burst Restoration and Enhancement [75.25451566988565]
  We propose a novel Gated Multi-Resolution Transfer Network (GMTNet) to reconstruct a spatially precise high-quality image from a burst of low-quality raw images. Detailed experimental analysis on five datasets validates our approach and sets a state-of-the-art for burst super-resolution, burst denoising, and low-light burst enhancement.
  arXiv  Detail & Related papers  (2023-04-13T17:54:00Z)
• Burst Image Restoration and Enhancement [86.08546447144377]
  The goal of Burst Image Restoration is to effectively combine complimentary cues across multiple burst frames to generate high-quality outputs. We create a set of emphpseudo-burst features that combine complimentary information from all the input burst frames to seamlessly exchange information. Our approach delivers state of the art performance on burst super-resolution and low-light image enhancement tasks.
  arXiv  Detail & Related papers  (2021-10-07T17:58:56Z)
• ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring [92.40655035360729]
  Video deblurring models exploit consecutive frames to remove blurs from camera shakes and object motions. We propose a novel implicit method to learn spatial correspondence among blurry frames in the feature space. Our proposed method is evaluated on the widely-adopted DVD dataset, along with a newly collected High-Frame-Rate (1000 fps) dataset for Video Deblurring.
  arXiv  Detail & Related papers  (2021-03-07T04:33:13Z)
• Deep Burst Super-Resolution [165.90445859851448]
  We propose a novel architecture for the burst super-resolution task. Our network takes multiple noisy RAW images as input, and generates a denoised, super-resolved RGB image as output. In order to enable training and evaluation on real-world data, we additionally introduce the BurstSR dataset.
  arXiv  Detail & Related papers  (2021-01-26T18:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.