Frequency-Guided Fusion For RGB-Thermal Semantic Segmentation

• URL: http://arxiv.org/abs/2605.26273v1
• Date: Mon, 25 May 2026 18:57:22 GMT
• Title: Frequency-Guided Fusion For RGB-Thermal Semantic Segmentation
• Abstract summary: We propose a multi-modal fusion architecture built upon dual ConvNeXt V2 backbones that employs stage-wise, modality-adaptive fusion strategies.<n>Experiments on MFNet and PST900 demonstrate that our lightest variant achieves 61.73% and 86.24% mIoU, respectively, with only 35.43M parameters.
• Score: 0.6187780920448871
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic segmentation in complex environments such as urban driving scenes remains challenging under adverse lighting conditions, where RGB images alone provide insufficient information. RGB-Thermal fusion leverages the complementary strengths of visible and infrared imagery to improve scene understanding; however, effectively integrating these heterogeneous modalities at varying levels of feature abstraction remains an open problem. In this paper, we propose a multi-modal fusion architecture built upon dual ConvNeXt V2 backbones that employs stage-wise, modality-adaptive fusion strategies. For early-stage features, we introduce a Frequency-Based Fusion Module that decomposes infrared features into low- and high-frequency components via Gaussian filtering, applies dual-branch spatial attention to selectively emphasize thermal patterns and fine-grained boundaries, and integrates them with RGB features through a confidence-gated residual mechanism. For late-stage features, we design a semantic fusion module with cross-modal attention and multi-scale depthwise convolutions to capture semantic correspondences across modalities. The fused features are decoded via a PANet-style bidirectional decoder with deep supervision. Experiments on MFNet and PST900 demonstrate that our lightest variant achieves 61.73\% and 86.24\% mIoU, respectively, with only 35.43M parameters, outperforming recent methods while using substantially fewer parameters and lower computational cost. Code is available at https://github.com/ismailemrecntz/VISIBLE-INFRARED-SENSOR-FUSION

Related papers

• SWIR-LightFusion: Multi-spectral Semantic Fusion of Synthetic SWIR with Thermal IR (LWIR/MWIR) and RGB [6.054759773528757]
  We propose a synthetically generate SWIR-like structural/contrast cues without claiming spectral reproduction.<n>We then propose a multimodal fusion framework integrating synthetic SWIR, LWIR, and RGB modalities.<n>The outcomes highlight substantial potential for real-world applications in surveillance and autonomous systems.
  arXiv  Detail & Related papers  (2025-10-15T11:00:41Z)
• HyPSAM: Hybrid Prompt-driven Segment Anything Model for RGB-Thermal Salient Object Detection [75.406055413928]
  We propose a novel prompt-driven segment anything model (HyPSAM) for RGB-T SOD.<n> DFNet employs dynamic convolution and multi-branch decoding to facilitate adaptive cross-modality interaction.<n>Experiments on three public datasets demonstrate that our method achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-09-23T07:32:11Z)
• TUNI: Real-time RGB-T Semantic Segmentation with Unified Multi-Modal Feature Extraction and Cross-Modal Feature Fusion [11.878642970457646]
  RGB-thermal (RGB-T) semantic segmentation improves the environmental perception of autonomous platforms in challenging conditions.<n>Prevailing models employ encoders pre-trained on RGB images to extract features from both RGB and infrared inputs.<n>We propose TUNI, with an RGB-T encoder consisting of multiple stacked blocks that simultaneously perform multi-modal feature extraction and cross-modal fusion.
  arXiv  Detail & Related papers  (2025-09-12T07:02:45Z)
• Spectral-Aware Global Fusion for RGB-Thermal Semantic Segmentation [10.761216101789774]
  We propose the Spectral-aware Global Fusion Network (SGFNet) to enhance and fuse the multi-modal features.<n>SGFNet outperforms the state-of-the-art methods on the MFNet and PST900 datasets.
  arXiv  Detail & Related papers  (2025-05-21T13:17:57Z)
• Bringing RGB and IR Together: Hierarchical Multi-Modal Enhancement for Robust Transmission Line Detection [67.02804741856512]
  We propose a novel Hierarchical Multi-Modal Enhancement Network (HMMEN) that integrates RGB and IR data for robust and accurate TL detection.<n>Our method introduces two key components: (1) a Mutual Multi-Modal Enhanced Block (MMEB), which fuses and enhances hierarchical RGB and IR feature maps in a coarse-to-fine manner, and (2) a Feature Alignment Block (FAB) that corrects misalignments between decoder outputs and IR feature maps by leveraging deformable convolutions.
  arXiv  Detail & Related papers  (2025-01-25T06:21:06Z)
• Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation [19.41334573257174]
  Traditional methods mostly use RGB images which are heavily affected by lighting conditions, eg, darkness. Recent studies show thermal images are robust to the night scenario as a compensating modality for segmentation. This work proposes a Residual Spatial Fusion Network (RSFNet) for RGB-T semantic segmentation.
  arXiv  Detail & Related papers  (2023-06-17T14:28:08Z)
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion [138.40422469153145]
  We propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. We show that CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion.
  arXiv  Detail & Related papers  (2022-11-26T02:40:28Z)
• Mirror Complementary Transformer Network for RGB-thermal Salient Object Detection [16.64781797503128]
  RGB-thermal object detection (RGB-T SOD) aims to locate the common prominent objects of an aligned visible and thermal infrared image pair. In this paper, we propose a novel mirror complementary Transformer network (MCNet) for RGB-T SOD. Experiments on benchmark and VT723 datasets show that the proposed method outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-07-07T20:26:09Z)
• Transformer-based Network for RGB-D Saliency Detection [82.6665619584628]
  Key to RGB-D saliency detection is to fully mine and fuse information at multiple scales across the two modalities. We show that transformer is a uniform operation which presents great efficacy in both feature fusion and feature enhancement. Our proposed network performs favorably against state-of-the-art RGB-D saliency detection methods.
  arXiv  Detail & Related papers  (2021-12-01T15:53:58Z)
• Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection [78.47767202232298]
  We propose a novel Cross-modality Discrepant Interaction Network (CDINet) for RGB-D SOD. Two components are designed to implement the effective cross-modality interaction. Our network outperforms $15$ state-of-the-art methods both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2021-08-04T11:24:42Z)
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation [59.94819184452694]
  Depth information has proven to be a useful cue in the semantic segmentation of RGBD images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion. In this paper, we propose a unified and efficient Crossmodality Guided to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively.
  arXiv  Detail & Related papers  (2020-07-17T18:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.