Efficient and Accurate Hyperspectral Pansharpening Using 3D VolumeNet and 2.5D Texture Transfer

• URL: http://arxiv.org/abs/2203.03951v1
• Date: Tue, 8 Mar 2022 09:24:12 GMT
• Title: Efficient and Accurate Hyperspectral Pansharpening Using 3D VolumeNet and 2.5D Texture Transfer
• Authors: Yinao Li, Yutaro Iwamoto, Ryousuke Nakamura, Lanfen Lin, Ruofeng Tong, Yen-Wei Chen
• Abstract summary: We propose a novel multi-spectral image fusion method using a combination of the previously proposed 3D CNN model VolumeNet and 2.5D texture transfer method. The experimental results show that the proposed method outperforms the existing methods in terms of objective accuracy assessment, method efficiency, and visual subjective evaluation.
• Score: 13.854539265252201
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, convolutional neural networks (CNN) have obtained promising results in single-image SR for hyperspectral pansharpening. However, enhancing CNNs' representation ability with fewer parameters and a shorter prediction time is a challenging and critical task. In this paper, we propose a novel multi-spectral image fusion method using a combination of the previously proposed 3D CNN model VolumeNet and 2.5D texture transfer method using other modality high resolution (HR) images. Since a multi-spectral (MS) image consists of several bands and each band is a 2D image slice, MS images can be seen as 3D data. Thus, we use the previously proposed VolumeNet to fuse HR panchromatic (PAN) images and bicubic interpolated MS images. Because the proposed 3D VolumeNet can effectively improve the accuracy by expanding the receptive field of the model, and due to its lightweight structure, we can achieve better performance against the existing method without purchasing a large number of remote sensing images for training. In addition, VolumeNet can restore the high-frequency information lost in the HR MR image as much as possible, reducing the difficulty of feature extraction in the following step: 2.5D texture transfer. As one of the latest technologies, deep learning-based texture transfer has been demonstrated to effectively and efficiently improve the visual performance and quality evaluation indicators of image reconstruction. Different from the texture transfer processing of RGB image, we use HR PAN images as the reference images and perform texture transfer for each frequency band of MS images, which is named 2.5D texture transfer. The experimental results show that the proposed method outperforms the existing methods in terms of objective accuracy assessment, method efficiency, and visual subjective evaluation.

Related papers

• GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement [51.97726804507328]
  We propose a novel approach for 3D mesh reconstruction from multi-view images. Our method takes inspiration from large reconstruction models that use a transformer-based triplane generator and a Neural Radiance Field (NeRF) model trained on multi-view images.
  arXiv  Detail & Related papers  (2024-06-09T05:19:24Z)
• TP3M: Transformer-based Pseudo 3D Image Matching with Reference Image [0.9831489366502301]
  We propose a Transformer-based pseudo 3D image matching method. It upgrades the 2D features extracted from the source image to 3D features with the help of a reference image and matches to the 2D features extracted from the destination image. Experimental results on multiple datasets show that the proposed method achieves the state-of-the-art on the tasks of homography estimation, pose estimation and visual localization.
  arXiv  Detail & Related papers  (2024-05-14T08:56:09Z)
• Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object Structure via HyperNetworks [53.67497327319569]
  We introduce a novel neural rendering technique to solve image-to-3D from a single view. Our approach employs the signed distance function as the surface representation and incorporates generalizable priors through geometry-encoding volumes and HyperNetworks. Our experiments show the advantages of our proposed approach with consistent results and rapid generation.
  arXiv  Detail & Related papers  (2023-12-24T08:42:37Z)
• Wonder3D: Single Image to 3D using Cross-Domain Diffusion [105.16622018766236]
  Wonder3D is a novel method for efficiently generating high-fidelity textured meshes from single-view images. To holistically improve the quality, consistency, and efficiency of image-to-3D tasks, we propose a cross-domain diffusion model.
  arXiv  Detail & Related papers  (2023-10-23T15:02:23Z)
• Guide3D: Create 3D Avatars from Text and Image Guidance [55.71306021041785]
  Guide3D is a text-and-image-guided generative model for 3D avatar generation based on diffusion models. Our framework produces topologically and structurally correct geometry and high-resolution textures.
  arXiv  Detail & Related papers  (2023-08-18T17:55:47Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• DH-GAN: A Physics-driven Untrained Generative Adversarial Network for 3D Microscopic Imaging using Digital Holography [3.4635026053111484]
  Digital holography is a 3D imaging technique by emitting a laser beam with a plane wavefront to an object and measuring the intensity of the diffracted waveform, called holograms. Recently, deep learning (DL) methods have been used for more accurate holographic processing. We propose a new DL architecture based on generative adversarial networks that uses a discriminative network for realizing a semantic measure for reconstruction quality.
  arXiv  Detail & Related papers  (2022-05-25T17:13:45Z)
• 3D Human Texture Estimation from a Single Image with Transformers [106.6320286821364]
  We propose a Transformer-based framework for 3D human texture estimation from a single image. We also propose a mask-fusion strategy to combine the advantages of the RGB-based and texture-flow-based models.
  arXiv  Detail & Related papers  (2021-09-06T16:00:20Z)
• Dense Dilated UNet: Deep Learning for 3D Photoacoustic Tomography Image Reconstruction [1.5749416770494706]
  We propose a modified convolutional neural network (CNN) architecture termed Dense Dilation UNet (DD-UNet) for correcting artifacts in 3D photoacoustic tomography (PAT) We compare the proposed CNN in terms of image quality as measured by the multiscale structural similarity index metric to the Fully Dense UNet (FD-UNet) Results demonstrate that the DD-Net consistently outperforms the FD-UNet and is able to more reliably reconstruct smaller image features.
  arXiv  Detail & Related papers  (2021-04-07T14:01:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.