Multimodal Deep Unfolding for Guided Image Super-Resolution
- URL: http://arxiv.org/abs/2001.07575v1
- Date: Tue, 21 Jan 2020 14:41:53 GMT
- Title: Multimodal Deep Unfolding for Guided Image Super-Resolution
- Authors: Iman Marivani, Evaggelia Tsiligianni, Bruno Cornelis and Nikos
Deligiannis
- Abstract summary: Deep learning methods rely on training data to learn an end-to-end mapping from a low-resolution input to a high-resolution output.
We propose a multimodal deep learning design that incorporates sparse priors and allows the effective integration of information from another image modality into the network architecture.
Our solution relies on a novel deep unfolding operator, performing steps similar to an iterative algorithm for convolutional sparse coding with side information.
- Score: 23.48305854574444
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The reconstruction of a high resolution image given a low resolution
observation is an ill-posed inverse problem in imaging. Deep learning methods
rely on training data to learn an end-to-end mapping from a low-resolution
input to a high-resolution output. Unlike existing deep multimodal models that
do not incorporate domain knowledge about the problem, we propose a multimodal
deep learning design that incorporates sparse priors and allows the effective
integration of information from another image modality into the network
architecture. Our solution relies on a novel deep unfolding operator,
performing steps similar to an iterative algorithm for convolutional sparse
coding with side information; therefore, the proposed neural network is
interpretable by design. The deep unfolding architecture is used as a core
component of a multimodal framework for guided image super-resolution. An
alternative multimodal design is investigated by employing residual learning to
improve the training efficiency. The presented multimodal approach is applied
to super-resolution of near-infrared and multi-spectral images as well as depth
upsampling using RGB images as side information. Experimental results show that
our model outperforms state-of-the-art methods.
Related papers
- Pixel-Aligned Multi-View Generation with Depth Guided Decoder [86.1813201212539]
We propose a novel method for pixel-level image-to-multi-view generation.
Unlike prior work, we incorporate attention layers across multi-view images in the VAE decoder of a latent video diffusion model.
Our model enables better pixel alignment across multi-view images.
arXiv Detail & Related papers (2024-08-26T04:56:41Z) - Research on Image Super-Resolution Reconstruction Mechanism based on Convolutional Neural Network [8.739451985459638]
Super-resolution algorithms transform one or more sets of low-resolution images captured from the same scene into high-resolution images.
The extraction of image features and nonlinear mapping methods in the reconstruction process remain challenging for existing algorithms.
The objective is to recover high-quality, high-resolution images from low-resolution images.
arXiv Detail & Related papers (2024-07-18T06:50:39Z) - Learning Enriched Features for Fast Image Restoration and Enhancement [166.17296369600774]
This paper presents a holistic goal of maintaining spatially-precise high-resolution representations through the entire network.
We learn an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details.
Our approach achieves state-of-the-art results for a variety of image processing tasks, including defocus deblurring, image denoising, super-resolution, and image enhancement.
arXiv Detail & Related papers (2022-04-19T17:59:45Z) - Deep Reparametrization of Multi-Frame Super-Resolution and Denoising [167.42453826365434]
We propose a deep reparametrization of the maximum a posteriori formulation commonly employed in multi-frame image restoration tasks.
Our approach is derived by introducing a learned error metric and a latent representation of the target image.
We validate our approach through comprehensive experiments on burst denoising and burst super-resolution datasets.
arXiv Detail & Related papers (2021-08-18T17:57:02Z) - High-Frequency aware Perceptual Image Enhancement [0.08460698440162888]
We introduce a novel deep neural network suitable for multi-scale analysis and propose efficient model-agnostic methods.
Our model can be applied to multi-scale image enhancement problems including denoising, deblurring and single image super-resolution.
arXiv Detail & Related papers (2021-05-25T07:33:14Z) - High-resolution Depth Maps Imaging via Attention-based Hierarchical
Multi-modal Fusion [84.24973877109181]
We propose a novel attention-based hierarchical multi-modal fusion network for guided DSR.
We show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.
arXiv Detail & Related papers (2021-04-04T03:28:33Z) - Interpretable Deep Multimodal Image Super-Resolution [23.48305854574444]
Multimodal image super-resolution (SR) is the reconstruction of a high resolution image given a low-resolution observation with the aid of another image modality.
We present a multimodal deep network design that integrates coupled sparse priors and allows the effective fusion of information from another modality into the reconstruction process.
arXiv Detail & Related papers (2020-09-07T14:08:35Z) - Deep Iterative Residual Convolutional Network for Single Image
Super-Resolution [31.934084942626257]
We propose a deep Iterative Super-Resolution Residual Convolutional Network (ISRResCNet)
It exploits the powerful image regularization and large-scale optimization techniques by training the deep network in an iterative manner with a residual learning approach.
Our method with a few trainable parameters improves the results for different scaling factors in comparison with the state-of-art methods.
arXiv Detail & Related papers (2020-09-07T12:54:14Z) - Learning Enriched Features for Real Image Restoration and Enhancement [166.17296369600774]
convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task.
We present a novel architecture with the collective goals of maintaining spatially-precise high-resolution representations through the entire network.
Our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details.
arXiv Detail & Related papers (2020-03-15T11:04:30Z) - Learned Multi-View Texture Super-Resolution [76.82725815863711]
We present a super-resolution method capable of creating a high-resolution texture map for a virtual 3D object from a set of lower-resolution images of that object.
Our architecture unifies the concepts of (i) multi-view super-resolution based on the redundancy of overlapping views and (ii) single-view super-resolution based on a learned prior of high-resolution (HR) image structure.
arXiv Detail & Related papers (2020-01-14T13:49:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.