Balancing Task-invariant Interaction and Task-specific Adaptation for Unified Image Fusion

• URL: http://arxiv.org/abs/2504.05164v1
• Date: Mon, 07 Apr 2025 15:08:35 GMT
• Title: Balancing Task-invariant Interaction and Task-specific Adaptation for Unified Image Fusion
• Authors: Xingyu Hu, Junjun Jiang, Chenyang Wang, Kui Jiang, Xianming Liu, Jiayi Ma,
• Abstract summary: Unified image fusion aims to integrate complementary information from multi-source images, enhancing image quality.<n>Existing general image fusion methods incorporate explicit task identification to enable adaptation to different fusion tasks.<n>We propose a novel unified image fusion framework named "TITA", which balances Task-invariant Interaction and Task-specific Adaptation.
• Score: 82.74585945197231
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unified image fusion aims to integrate complementary information from multi-source images, enhancing image quality through a unified framework applicable to diverse fusion tasks. While treating all fusion tasks as a unified problem facilitates task-invariant knowledge sharing, it often overlooks task-specific characteristics, thereby limiting the overall performance. Existing general image fusion methods incorporate explicit task identification to enable adaptation to different fusion tasks. However, this dependence during inference restricts the model's generalization to unseen fusion tasks. To address these issues, we propose a novel unified image fusion framework named "TITA", which dynamically balances both Task-invariant Interaction and Task-specific Adaptation. For task-invariant interaction, we introduce the Interaction-enhanced Pixel Attention (IPA) module to enhance pixel-wise interactions for better multi-source complementary information extraction. For task-specific adaptation, the Operation-based Adaptive Fusion (OAF) module dynamically adjusts operation weights based on task properties. Additionally, we incorporate the Fast Adaptive Multitask Optimization (FAMO) strategy to mitigate the impact of gradient conflicts across tasks during joint training. Extensive experiments demonstrate that TITA not only achieves competitive performance compared to specialized methods across three image fusion scenarios but also exhibits strong generalization to unseen fusion tasks.

Related papers

• One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion [38.16599550115468]
  We propose to leverage low-level vision tasks from digital photography fusion, allowing for effective feature interaction through pixel-level supervision.<n>The proposed GIFNet supports diverse fusion tasks, achieving high performance across both seen and unseen scenarios with a single model.
  arXiv  Detail & Related papers  (2025-02-27T07:55:19Z)
• Instruction-Driven Fusion of Infrared-Visible Images: Tailoring for Diverse Downstream Tasks [9.415977819944246]
  The primary value of infrared and visible image fusion technology lies in applying the fusion results to downstream tasks. Existing methods face challenges such as increased training complexity and significantly compromised performance of individual tasks. We propose Task-Oriented Adaptive Regulation (T-OAR), an adaptive mechanism specifically designed for multi-task environments.
  arXiv  Detail & Related papers  (2024-11-14T12:02:01Z)
• Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond [74.96466744512992]
  The essence of image fusion is to integrate complementary information from source images. DeFusion++ produces versatile fused representations that can enhance the quality of image fusion and the effectiveness of downstream high-level vision tasks.
  arXiv  Detail & Related papers  (2024-10-16T06:28:49Z)
• Task-Customized Mixture of Adapters for General Image Fusion [51.8742437521891]
  General image fusion aims at integrating important information from multi-source images. We propose a novel task-customized mixture of adapters (TC-MoA) for general image fusion, adaptively prompting various fusion tasks in a unified model.
  arXiv  Detail & Related papers  (2024-03-19T07:02:08Z)
• Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation [66.15246197473897]
  Multi-modality image fusion and segmentation play a vital role in autonomous driving and robotic operation. We propose a textbfMulti-textbfinteractive textbfFeature learning architecture for image fusion and textbfSegmentation.
  arXiv  Detail & Related papers  (2023-08-04T01:03:58Z)
• A Task-guided, Implicitly-searched and Meta-initialized Deep Model for Image Fusion [69.10255211811007]
  We present a Task-guided, Implicit-searched and Meta- generalizationd (TIM) deep model to address the image fusion problem in a challenging real-world scenario. Specifically, we propose a constrained strategy to incorporate information from downstream tasks to guide the unsupervised learning process of image fusion. Within this framework, we then design an implicit search scheme to automatically discover compact architectures for our fusion model with high efficiency.
  arXiv  Detail & Related papers  (2023-05-25T08:54:08Z)
• Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyond [50.556961575275345]
  We build an image fusion module to fuse complementary characteristics and cascade dual task-related modules. We develop an efficient first-order approximation to compute corresponding gradients and present dynamic weighted aggregation to balance the gradients for fusion learning.
  arXiv  Detail & Related papers  (2023-05-11T10:55:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.