Text-Driven Fusion for Infrared and Visible Images: Achieving Image Scene Adaptation on Hyperbolic Space

• URL: http://arxiv.org/abs/2606.15104v1
• Date: Sat, 13 Jun 2026 04:33:33 GMT
• Title: Text-Driven Fusion for Infrared and Visible Images: Achieving Image Scene Adaptation on Hyperbolic Space
• Abstract summary: Infrared and visible image fusion aims to integrate complementary modalities.<n>We introduce a text-driven fusion framework empowered by hyperbolic manifold learning.<n> Experimental results show our method outperforms state-of-the-art approaches on benchmark datasets.
• Score: 63.220767051340005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Infrared and visible image fusion aims to integrate complementary modalities, while existing Euclidean methods impose rigid distance metrics that distort multi-modal interactions and parent-to-child semantic hierarchies. To overcome these limitations, we introduce a text-driven fusion framework empowered by hyperbolic manifold learning. During training, BLIP-extracted text prompts serve as topological anchors within the hyperbolic space, guiding vision-attribute alignment through hyperbolic embeddings that naturally accommodate varying semantic granularities. By exploiting the exponential volume growth dictated by the Poincaré ball's negative curvature, this approach seamlessly embeds hierarchical trees to encode coarse-to-fine semantics without metric saturation, while the vast peripheral space prevents texture distortion during cross-modal fusion. At inference, the fusion process autonomously adapts to input content using the learned text-attribute priors, completely eliminating the need for textual input. Experimental results show our method outperforms state-of-the-art approaches on benchmark datasets, with code available at https://github.com/Shaoyun2023/TEDFusion.

Related papers

• IT-TextFusion: Iterative Text-Image Interaction with Text-Guided Residual Refinement for Degradation-Aware Image Fusion [2.0260533411307304]
  We propose an iterative text-guided image fusion framework that incorporates text-conditioned feature interaction across multiple fusion and refinement stages.<n>The proposed method integrates deepest-level Cross-Attention, multi-scale Cross-Gate Fusion, and stage-specific text-conditioned modulation.<n> Experiments on several benchmark datasets show that the proposed method improves several information-preservation and perceptual-quality metrics.
  arXiv  Detail & Related papers  (2026-09-01T11:33:20Z)
• EAPFusion: Intrinsic Evolving Auxiliary Prior Guidance for Infrared and Visible Image Fusion [15.580984590527343]
  Infrared-visible image fusion aims to create an information-rich fused image by integrating the complementary thermal saliency from infrared sensing and fine textures from visible imaging.<n>Most existing fusion methods rely upon static trained weights that cannot adapt to scene-specific content at inference time.<n>We propose EAPFusion to address these issues by using self-evolving intrinsic priors instead of relying on external auxiliary models.
  arXiv  Detail & Related papers  (2026-05-03T14:56:00Z)
• CtrlFuse: Mask-Prompt Guided Controllable Infrared and Visible Image Fusion [51.060328159429154]
  Infrared and visible image fusion generates all-weather perception-capable images by combining complementary modalities.<n>We propose CtrlFuse, a controllable image fusion framework that enables interactive dynamic fusion guided by mask prompts.<n> Experiments demonstrate state-of-the-art results in both fusion controllability and segmentation accuracy, with the adapted task branch even outperforming the original segmentation model.
  arXiv  Detail & Related papers  (2026-01-12T13:36:48Z)
• Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds [49.95082206008502]
  Alignment across Trees is a method that constructs and aligns tree-like hierarchical features for both image and text modalities.<n>We introduce a semantic-aware visual feature extraction framework that applies a cross-attention mechanism to visual class tokens from intermediate Transformer layers.
  arXiv  Detail & Related papers  (2025-10-31T11:32:15Z)
• Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers [56.76198904599581]
  Text-to-image diffusion models excel at translating language prompts into implicitly grounding concepts through their cross-modal attention mechanisms.<n>Recent multi-modal diffusion transformers extend this by introducing joint self-attentiond image and text tokens, enabling richer and more scalable cross-modal alignment.<n>We introduce Seg4Diff, a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image.
  arXiv  Detail & Related papers  (2025-09-22T17:59:54Z)
• MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusion [10.160499805076755]
  We introduce MSGFusion, a multimodal scene graph-guided fusion framework for infrared and visible imagery.<n>By deeply coupling structured scene graphs derived from text and vision, MSGFusion explicitly represents entities, attributes, and spatial relations.<n>It delivers superior semantic consistency and generalizability in downstream tasks such as low-light object detection, semantic segmentation, and medical image fusion.
  arXiv  Detail & Related papers  (2025-09-16T09:58:06Z)
• GrFormer: A Novel Transformer on Grassmann Manifold for Infrared and Visible Image Fusion [33.925249998725896]
  We propose a novel attention mechanism based on Grassmann manifold for infrared and visible image fusion.<n>Our method constructs a low-rank subspace mapping through projection constraints on the Grassmann manifold.<n>This forces the features to decouple into high-frequency details (local low-rank) and low-frequency semantics (global low-rank)
  arXiv  Detail & Related papers  (2025-06-17T10:32:05Z)
• ShapeShift: Towards Text-to-Shape Arrangement Synthesis with Content-Aware Geometric Constraints [13.2441524021269]
  ShapeShift is a text-guided image-to-image translation task that requires rearranging the input set of rigid shapes into non-overlapping configurations.<n>We introduce a content-aware collision resolution mechanism that applies minimal semantically coherent adjustments when overlaps occur.<n>Our approach yields interpretable compositions where spatial relationships clearly embody the textual prompt.
  arXiv  Detail & Related papers  (2025-03-18T20:48:58Z)
• From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion [66.33467192279514]
  We introduce a text-guided multi-modality image fusion method that leverages the high-level semantics from textual descriptions to integrate semantics from infrared and visible images. Our method not only produces visually superior fusion results but also achieves a higher detection mAP over existing methods, achieving state-of-the-art results.
  arXiv  Detail & Related papers  (2023-12-31T08:13:47Z)
• Fine-Grained Semantically Aligned Vision-Language Pre-Training [151.7372197904064]
  Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks. Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts. We introduce LO, a fine-grained semantically aLigned visiOn-langUage PrE-training framework, which learns fine-grained semantic alignment from the novel perspective of game-theoretic interactions.
  arXiv  Detail & Related papers  (2022-08-04T07:51:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.