ReFusion: Learning Image Fusion from Reconstruction with Learnable Loss
via Meta-Learning
- URL: http://arxiv.org/abs/2312.07943v2
- Date: Mon, 11 Mar 2024 07:25:26 GMT
- Title: ReFusion: Learning Image Fusion from Reconstruction with Learnable Loss
via Meta-Learning
- Authors: Haowen Bai, Zixiang Zhao, Jiangshe Zhang, Yichen Wu, Lilun Deng, Yukun
Cui, Shuang Xu, Baisong Jiang
- Abstract summary: We introduce a unified image fusion framework based on meta-learning, named ReFusion.
ReFusion employs a parameterized loss function, dynamically adjusted by the training framework according to the specific scenario and task.
It is capable of adapting to various tasks, including infrared-visible, medical, multi-focus, and multi-exposure image fusion.
- Score: 17.91346343984845
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Image fusion aims to combine information from multiple source images into a
single one with more comprehensive informational content. The significant
challenges for deep learning-based image fusion algorithms are the lack of a
definitive ground truth as well as the corresponding distance measurement, with
current manually given loss functions constrain the flexibility of model and
generalizability for unified fusion tasks. To overcome these limitations, we
introduce a unified image fusion framework based on meta-learning, named
ReFusion, which provides a learning paradigm that obtains the optimal fusion
loss for various fusion tasks based on reconstructing the source images.
Compared to existing methods, ReFusion employs a parameterized loss function,
dynamically adjusted by the training framework according to the specific
scenario and task. ReFusion is constituted by three components: a fusion
module, a loss proposal module, and a source reconstruction module. To ensure
the fusion module maximally preserves the information from the source images,
enabling the reconstruction of the source images from the fused image, we adopt
a meta-learning strategy to train the loss proposal module using reconstruction
loss. The update of the fusion module relies on the fusion loss proposed by the
loss proposal module. The alternating updates of the three modules mutually
facilitate each other, aiming to propose an appropriate fusion loss for
different tasks and yield satisfactory fusion results. Extensive experiments
demonstrate that ReFusion is capable of adapting to various tasks, including
infrared-visible, medical, multi-focus, and multi-exposure image fusion. The
code will be released.
Related papers
- Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model [30.739879255847946]
Existing multi-modal image fusion methods fail to address the compound degradations presented in source images.
This study proposes a novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse.
arXiv Detail & Related papers (2024-10-31T13:10:50Z) - Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond [74.96466744512992]
The essence of image fusion is to integrate complementary information from source images.
DeFusion++ produces versatile fused representations that can enhance the quality of image fusion and the effectiveness of downstream high-level vision tasks.
arXiv Detail & Related papers (2024-10-16T06:28:49Z) - A Task-guided, Implicitly-searched and Meta-initialized Deep Model for
Image Fusion [69.10255211811007]
We present a Task-guided, Implicit-searched and Meta- generalizationd (TIM) deep model to address the image fusion problem in a challenging real-world scenario.
Specifically, we propose a constrained strategy to incorporate information from downstream tasks to guide the unsupervised learning process of image fusion.
Within this framework, we then design an implicit search scheme to automatically discover compact architectures for our fusion model with high efficiency.
arXiv Detail & Related papers (2023-05-25T08:54:08Z) - Searching a Compact Architecture for Robust Multi-Exposure Image Fusion [55.37210629454589]
Two major stumbling blocks hinder the development, including pixel misalignment and inefficient inference.
This study introduces an architecture search-based paradigm incorporating self-alignment and detail repletion modules for robust multi-exposure image fusion.
The proposed method outperforms various competitive schemes, achieving a noteworthy 3.19% improvement in PSNR for general scenarios and an impressive 23.5% enhancement in misaligned scenarios.
arXiv Detail & Related papers (2023-05-20T17:01:52Z) - Equivariant Multi-Modality Image Fusion [124.11300001864579]
We propose the Equivariant Multi-Modality imAge fusion paradigm for end-to-end self-supervised learning.
Our approach is rooted in the prior knowledge that natural imaging responses are equivariant to certain transformations.
Experiments confirm that EMMA yields high-quality fusion results for infrared-visible and medical images.
arXiv Detail & Related papers (2023-05-19T05:50:24Z) - LRRNet: A Novel Representation Learning Guided Fusion Network for
Infrared and Visible Images [98.36300655482196]
We formulate the fusion task mathematically, and establish a connection between its optimal solution and the network architecture that can implement it.
In particular we adopt a learnable representation approach to the fusion task, in which the construction of the fusion network architecture is guided by the optimisation algorithm producing the learnable model.
Based on this novel network architecture, an end-to-end lightweight fusion network is constructed to fuse infrared and visible light images.
arXiv Detail & Related papers (2023-04-11T12:11:23Z) - CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature
Ensemble for Multi-modality Image Fusion [72.8898811120795]
We propose a coupled contrastive learning network, dubbed CoCoNet, to realize infrared and visible image fusion.
Our method achieves state-of-the-art (SOTA) performance under both subjective and objective evaluation.
arXiv Detail & Related papers (2022-11-20T12:02:07Z) - TransFuse: A Unified Transformer-based Image Fusion Framework using
Self-supervised Learning [5.849513679510834]
Image fusion is a technique to integrate information from multiple source images with complementary information to improve the richness of a single image.
Two-stage methods avoid the need of large amount of task-specific training data by training encoder-decoder network on large natural image datasets.
We propose a destruction-reconstruction based self-supervised training scheme to encourage the network to learn task-specific features.
arXiv Detail & Related papers (2022-01-19T07:30:44Z) - RFN-Nest: An end-to-end residual fusion network for infrared and visible
images [37.935940961760785]
We propose an end-to-end fusion network architecture (RFN-Nest) for infrared and visible image fusion.
A novel detail-preserving loss function, and a feature enhancing loss function are proposed to train RFN.
The experimental results on public domain data sets show that, compared with the existing methods, our end-to-end fusion network delivers a better performance than the state-of-the-art methods.
arXiv Detail & Related papers (2021-03-07T07:29:50Z) - UFA-FUSE: A novel deep supervised and hybrid model for multi-focus image
fusion [4.105749631623888]
Traditional and deep learning-based fusion methods generate the intermediate decision map through a series of post-processing procedures.
Inspired by the image reconstruction techniques based on deep learning, we propose a multi-focus image fusion network framework.
We show that the proposed approach for multi-focus image fusion achieves remarkable fusion performance compared to 19 state-of-the-art fusion methods.
arXiv Detail & Related papers (2021-01-12T14:33:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.