Interpretable Deep Multimodal Image Super-Resolution

• URL: http://arxiv.org/abs/2009.03118v1
• Date: Mon, 7 Sep 2020 14:08:35 GMT
• Title: Interpretable Deep Multimodal Image Super-Resolution
• Authors: Iman Marivani, Evaggelia Tsiligianni, Bruno Cornelis, Nikos Deligiannis
• Abstract summary: Multimodal image super-resolution (SR) is the reconstruction of a high resolution image given a low-resolution observation with the aid of another image modality. We present a multimodal deep network design that integrates coupled sparse priors and allows the effective fusion of information from another modality into the reconstruction process.
• Score: 23.48305854574444
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal image super-resolution (SR) is the reconstruction of a high resolution image given a low-resolution observation with the aid of another image modality. While existing deep multimodal models do not incorporate domain knowledge about image SR, we present a multimodal deep network design that integrates coupled sparse priors and allows the effective fusion of information from another modality into the reconstruction process. Our method is inspired by a novel iterative algorithm for coupled convolutional sparse coding, resulting in an interpretable network by design. We apply our model to the super-resolution of near-infrared image guided by RGB images. Experimental results show that our model outperforms state-of-the-art methods.

Related papers

• DSR-Diff: Depth Map Super-Resolution with Diffusion Model [38.68563026759223]
  We present a novel CDSR paradigm that utilizes a diffusion model within the latent space to generate guidance for depth map super-resolution. Our proposed method has shown superior performance in extensive experiments when compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-11-16T14:18:10Z)
• Deep Unfolding Convolutional Dictionary Model for Multi-Contrast MRI Super-resolution and Reconstruction [23.779641808300596]
  We propose a multi-contrast convolutional dictionary (MC-CDic) model under the guidance of the optimization algorithm. We employ the proximal gradient algorithm to optimize the model and unroll the iterative steps into a deep CDic model. Experimental results demonstrate the superior performance of the proposed MC-CDic model against existing SOTA methods.
  arXiv  Detail & Related papers  (2023-09-03T13:18:59Z)
• Learning Enriched Features for Fast Image Restoration and Enhancement [166.17296369600774]
  This paper presents a holistic goal of maintaining spatially-precise high-resolution representations through the entire network. We learn an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Our approach achieves state-of-the-art results for a variety of image processing tasks, including defocus deblurring, image denoising, super-resolution, and image enhancement.
  arXiv  Detail & Related papers  (2022-04-19T17:59:45Z)
• Towards Reliable Image Outpainting: Learning Structure-Aware Multimodal Fusion with Depth Guidance [49.94504248096527]
  We propose a Depth-Guided Outpainting Network (DGONet) to model the feature representations of different modalities. Two components are designed to implement: 1) The Multimodal Learning Module produces unique depth and RGB feature representations from perspectives of different modal characteristics. We specially design an additional constraint strategy consisting of Cross-modal Loss and Edge Loss to enhance ambiguous contours and expedite reliable content generation.
  arXiv  Detail & Related papers  (2022-04-12T06:06:50Z)
• Multi-modal Aggregation Network for Fast MR Imaging [85.25000133194762]
  We propose a novel Multi-modal Aggregation Network, named MANet, which is capable of discovering complementary representations from a fully sampled auxiliary modality. In our MANet, the representations from the fully sampled auxiliary and undersampled target modalities are learned independently through a specific network. Our MANet follows a hybrid domain learning framework, which allows it to simultaneously recover the frequency signal in the $k$-space domain.
  arXiv  Detail & Related papers  (2021-10-15T13:16:59Z)
• High-Frequency aware Perceptual Image Enhancement [0.08460698440162888]
  We introduce a novel deep neural network suitable for multi-scale analysis and propose efficient model-agnostic methods. Our model can be applied to multi-scale image enhancement problems including denoising, deblurring and single image super-resolution.
  arXiv  Detail & Related papers  (2021-05-25T07:33:14Z)
• Accurate and Lightweight Image Super-Resolution with Model-Guided Deep Unfolding Network [63.69237156340457]
  We present and advocate an explainable approach toward SISR named model-guided deep unfolding network (MoG-DUN) MoG-DUN is accurate (producing fewer aliasing artifacts), computationally efficient (with reduced model parameters), and versatile (capable of handling multiple degradations) The superiority of the proposed MoG-DUN method to existing state-of-theart image methods including RCAN, SRDNF, and SRFBN is substantiated by extensive experiments on several popular datasets and various degradation scenarios.
  arXiv  Detail & Related papers  (2020-09-14T08:23:37Z)
• Learning Enriched Features for Real Image Restoration and Enhancement [166.17296369600774]
  convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. We present a novel architecture with the collective goals of maintaining spatially-precise high-resolution representations through the entire network. Our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details.
  arXiv  Detail & Related papers  (2020-03-15T11:04:30Z)
• Multimodal Deep Unfolding for Guided Image Super-Resolution [23.48305854574444]
  Deep learning methods rely on training data to learn an end-to-end mapping from a low-resolution input to a high-resolution output. We propose a multimodal deep learning design that incorporates sparse priors and allows the effective integration of information from another image modality into the network architecture. Our solution relies on a novel deep unfolding operator, performing steps similar to an iterative algorithm for convolutional sparse coding with side information.
  arXiv  Detail & Related papers  (2020-01-21T14:41:53Z)
• Learned Multi-View Texture Super-Resolution [76.82725815863711]
  We present a super-resolution method capable of creating a high-resolution texture map for a virtual 3D object from a set of lower-resolution images of that object. Our architecture unifies the concepts of (i) multi-view super-resolution based on the redundancy of overlapping views and (ii) single-view super-resolution based on a learned prior of high-resolution (HR) image structure.
  arXiv  Detail & Related papers  (2020-01-14T13:49:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.