Deep 3D World Models for Multi-Image Super-Resolution Beyond Optical Flow

• URL: http://arxiv.org/abs/2401.16972v1
• Date: Tue, 30 Jan 2024 12:55:49 GMT
• Title: Deep 3D World Models for Multi-Image Super-Resolution Beyond Optical Flow
• Authors: Luca Savant Aira, Diego Valsesia, Andrea Bordone Molini, Giulia Fracastoro, Enrico Magli, Andrea Mirabile
• Abstract summary: Multi-image super-resolution (MISR) allows to increase the spatial resolution of a low-resolution (LR) acquisition by combining multiple images. Our proposed model, called EpiMISR, moves away from optical flow and explicitly uses the epipolar geometry of the acquisition process.
• Score: 27.31768206943397
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-image super-resolution (MISR) allows to increase the spatial resolution of a low-resolution (LR) acquisition by combining multiple images carrying complementary information in the form of sub-pixel offsets in the scene sampling, and can be significantly more effective than its single-image counterpart. Its main difficulty lies in accurately registering and fusing the multi-image information. Currently studied settings, such as burst photography, typically involve assumptions of small geometric disparity between the LR images and rely on optical flow for image registration. We study a MISR method that can increase the resolution of sets of images acquired with arbitrary, and potentially wildly different, camera positions and orientations, generalizing the currently studied MISR settings. Our proposed model, called EpiMISR, moves away from optical flow and explicitly uses the epipolar geometry of the acquisition process, together with transformer-based processing of radiance feature fields to substantially improve over state-of-the-art MISR methods in presence of large disparities in the LR images.

Related papers

• ESC-MISR: Enhancing Spatial Correlations for Multi-Image Super-Resolution in Remote Sensing [9.782167510476599]
  Multi-Image Super-Resolution in Remote Sensing (MISR-RS) is a crucial yet challenging research task in the remote sensing community. We propose a novel framework named Enhancing Spatial Correlations in MISR (ESC-MISR), which fully exploits the spatial-temporal relations of multiple images for HR image reconstruction. Our method achieves 0.70dB and 0.76dB cPSNR improvements on the two bands of the PROBA-V dataset respectively.
  arXiv  Detail & Related papers  (2024-11-07T13:45:04Z)
• QMambaBSR: Burst Image Super-Resolution with Query State Space Model [55.56075874424194]
  Burst super-resolution aims to reconstruct high-resolution images with higher quality and richer details by fusing the sub-pixel information from multiple burst low-resolution frames. In BusrtSR, the key challenge lies in extracting the base frame's content complementary sub-pixel details while simultaneously suppressing high-frequency noise disturbance. We introduce a novel Query Mamba Burst Super-Resolution (QMambaBSR) network, which incorporates a Query State Space Model (QSSM) and Adaptive Up-sampling module (AdaUp)
  arXiv  Detail & Related papers  (2024-08-16T11:15:29Z)
• Enhancing Low-light Light Field Images with A Deep Compensation Unfolding Network [52.77569396659629]
  This paper presents the deep compensation network unfolding (DCUNet) for restoring light field (LF) images captured under low-light conditions. The framework uses the intermediate enhanced result to estimate the illumination map, which is then employed in the unfolding process to produce a new enhanced result. To properly leverage the unique characteristics of LF images, this paper proposes a pseudo-explicit feature interaction module.
  arXiv  Detail & Related papers  (2023-08-10T07:53:06Z)
• Real Image Super-Resolution using GAN through modeling of LR and HR process [20.537597542144916]
  We propose a learnable adaptive sinusoidal nonlinearities incorporated in LR and SR models by directly learn degradation distributions. We demonstrate the effectiveness of our proposed approach in quantitative and qualitative experiments.
  arXiv  Detail & Related papers  (2022-10-19T09:23:37Z)
• Reference-based Image Super-Resolution with Deformable Attention Transformer [62.71769634254654]
  RefSR aims to exploit auxiliary reference (Ref) images to super-resolve low-resolution (LR) images. This paper proposes a deformable attention Transformer, namely DATSR, with multiple scales. Experiments demonstrate that our DATSR achieves state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2022-07-25T07:07:00Z)
• Deep Burst Super-Resolution [165.90445859851448]
  We propose a novel architecture for the burst super-resolution task. Our network takes multiple noisy RAW images as input, and generates a denoised, super-resolved RGB image as output. In order to enable training and evaluation on real-world data, we additionally introduce the BurstSR dataset.
  arXiv  Detail & Related papers  (2021-01-26T18:57:21Z)
• Boosting Image Super-Resolution Via Fusion of Complementary Information Captured by Multi-Modal Sensors [21.264746234523678]
  Image Super-Resolution (SR) provides a promising technique to enhance the image quality of low-resolution optical sensors. In this paper, we attempt to leverage complementary information from a low-cost channel (visible/depth) to boost image quality of an expensive channel (thermal) using fewer parameters.
  arXiv  Detail & Related papers  (2020-12-07T02:15:28Z)
• Cross-MPI: Cross-scale Stereo for Image Super-Resolution using Multiplane Images [44.85260985973405]
  Cross-MPI is an end-to-end RefSR network composed of a novel plane-aware MPI mechanism, a multiscale guided upsampling module and a super-resolution synthesis and fusion module. Experimental results on both digitally synthesized and optical zoom cross-scale data show that the Cross-MPI framework can achieve superior performance against the existing RefSR methods.
  arXiv  Detail & Related papers  (2020-11-30T09:14:07Z)
• A Parallel Down-Up Fusion Network for Salient Object Detection in Optical Remote Sensing Images [82.87122287748791]
  We propose a novel Parallel Down-up Fusion network (PDF-Net) for salient object detection in optical remote sensing images (RSIs) It takes full advantage of the in-path low- and high-level features and cross-path multi-resolution features to distinguish diversely scaled salient objects and suppress the cluttered backgrounds. Experiments on the ORSSD dataset demonstrate that the proposed network is superior to the state-of-the-art approaches both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-10-02T05:27:57Z)
• Multi-Dimension Fusion Network for Light Field Spatial Super-Resolution using Dynamic Filters [23.82780431526054]
  We introduce a novel learning-based framework to improve the spatial resolution of light fields. Our reconstructed images also show sharp details and distinct lines in both sub-aperture images and epipolar plane images.
  arXiv  Detail & Related papers  (2020-08-26T09:05:07Z)
• PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models [77.32079593577821]
  PULSE (Photo Upsampling via Latent Space Exploration) generates high-resolution, realistic images at resolutions previously unseen in the literature. Our method outperforms state-of-the-art methods in perceptual quality at higher resolutions and scale factors than previously possible.
  arXiv  Detail & Related papers  (2020-03-08T16:44:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.