RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation

• URL: http://arxiv.org/abs/2603.09149v1
• Date: Tue, 10 Mar 2026 03:40:26 GMT
• Title: RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation
• Abstract summary: RGB-Thermal (RGB-T) semantic segmentation is essential for robotic systems operating in low-light environments.<n>Traditional approaches often overemphasize modality balance, resulting in limited robustness or severe performance when sensor signals are partially missing.
• Score: 0.2578242050187029
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: RGB-Thermal (RGB-T) semantic segmentation is essential for robotic systems operating in low-light or dark environments. However, traditional approaches often overemphasize modality balance, resulting in limited robustness and severe performance degradation when sensor signals are partially missing. Recent advances such as cross-modal knowledge distillation and modality-adaptive fine-tuning attempt to enhance cross-modal interaction, but they typically decouple modality fusion and modality adaptation, requiring multi-stage training with frozen models or teacher-student frameworks. We present RTFDNet, a three-branch encoder-decoder that unifies fusion and decoupling for robust RGB-T segmentation. Synergistic Feature Fusion (SFF) performs channel-wise gated exchange and lightweight spatial attention to inject complementary cues. Cross-Modal Decouple Regularization (CMDR) isolates modality-specific components from the fused representation and supervises unimodal decoders via stop-gradient targets. Region Decouple Regularization (RDR) enforces class-selective prediction consistency in confident regions while blocking gradients to the fusion branch. This feedback loop strengthens unimodal paths without degrading the fused stream, enabling efficient standalone inference at test time. Extensive experiments demonstrate the effectiveness of RTFDNet, showing consistent performance across varying modality conditions. Our implementation will be released to facilitate further research. Our source code are publicly available at https://github.com/curapima/RTFDNet.

Related papers

• Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing [55.13802890769086]
  We present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing.<n>To achieve profound modal synergy, Evita embeds a suite of intrinsic co-learning modules directly into every encoder layer.<n>Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time multimodal perception.
  arXiv  Detail & Related papers  (2026-07-10T06:52:09Z)
• Active Adversarial Perturbation-driven Associative Memory Retrieval for RGB-Event Visual Object Tracking [58.338848435042536]
  RGB-Event tracking improves localization by fusing RGB appearance textures and dense temporal motion cues from event sensors.<n>Real-world scenes suffer diverse structured signal degradations that hinder traditional multi-modal fusion.<n>We present a hierarchical perturbation and retrieval framework tailored for RGB-Event tracking with robustness against partial target missing and modal degradation.
  arXiv  Detail & Related papers  (2026-06-24T23:34:06Z)
• RACANet: Reliability-Aware Crowd Anchor Network for RGB-T Crowd Counting [15.536708520170095]
  RGB-Thermal (T) crowd counting aims to integrate visible-spectrum and thermal infrared information to improve the robustness of crowd density estimation in complex scenes.<n>This paper proposes a two-stage fusion framework, RACANet, a Reliability-Aware Crowd Anchor Network for RGB-T crowd counting.<n>Experiments conducted on two widely used benchmark datasets, RGBT-CC and Drone-RGBT, demonstrate that RACANet outperforms existing methods.
  arXiv  Detail & Related papers  (2026-04-27T14:38:49Z)
• SwiTrack: Tri-State Switch for Cross-Modal Object Tracking [74.15663758681849]
  Cross-modal object tracking (CMOT) is an emerging task that maintains target consistency while the video stream switches between different modalities.<n>We propose SwiTrack, a novel state-switching framework that redefines CMOT through the deployment of three specialized streams.
  arXiv  Detail & Related papers  (2025-11-20T10:52:54Z)
• TUNI: Real-time RGB-T Semantic Segmentation with Unified Multi-Modal Feature Extraction and Cross-Modal Feature Fusion [11.878642970457646]
  RGB-thermal (RGB-T) semantic segmentation improves the environmental perception of autonomous platforms in challenging conditions.<n>Prevailing models employ encoders pre-trained on RGB images to extract features from both RGB and infrared inputs.<n>We propose TUNI, with an RGB-T encoder consisting of multiple stacked blocks that simultaneously perform multi-modal feature extraction and cross-modal fusion.
  arXiv  Detail & Related papers  (2025-09-12T07:02:45Z)
• FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion [92.4205087439928]
  Image-event joint depth estimation methods leverage complementary modalities for robust perception, yet face challenges in generalizability.<n>We propose the Self-supervised Transfer (PST) and the FrequencyDe-coupled Fusion module (FreDF)<n>PST establishes cross-modal knowledge transfer through latent space alignment with image foundation models, effectively mitigating data scarcity.<n>FreDF explicitly decouples high-frequency edge features from low-frequency structural components, resolving modality-specific frequency mismatches.<n>This combined approach enables FUSE to construct a universal image-event that only requires lightweight decoder adaptation for target datasets.
  arXiv  Detail & Related papers  (2025-03-25T15:04:53Z)
• IRFusionFormer: Enhancing Pavement Crack Segmentation with RGB-T Fusion and Topological-Based Loss [0.0]
  We propose IRFusionFormer, a novel model for crack segmentation that effectively integrates RGB and thermal data.<n>Our method achieves state-of-the-art performance, with a Dice score of 90.01% and an IoU of 81.83%, significantly improving robustness and accuracy in varying environmental conditions.
  arXiv  Detail & Related papers  (2024-09-30T16:35:16Z)
• WCCNet: Wavelet-context Cooperative Network for Efficient Multispectral Pedestrian Detection [29.43086971740499]
  We propose a novel framework named Wavelet-context Cooperative Network (WCCNet)<n>WCCNet is able to differentially extract complementary features of different spectra with lower computational complexity.<n>We elaborately design the crossmodal rearranging fusion module (CMRF) to amplify the crossmodal reciprocal information.
  arXiv  Detail & Related papers  (2023-08-02T09:35:21Z)
• Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation [19.41334573257174]
  Traditional methods mostly use RGB images which are heavily affected by lighting conditions, eg, darkness. Recent studies show thermal images are robust to the night scenario as a compensating modality for segmentation. This work proposes a Residual Spatial Fusion Network (RSFNet) for RGB-T semantic segmentation.
  arXiv  Detail & Related papers  (2023-06-17T14:28:08Z)
• Complementary Random Masking for RGB-Thermal Semantic Segmentation [63.93784265195356]
  RGB-thermal semantic segmentation is a potential solution to achieve reliable semantic scene understanding in adverse weather and lighting conditions. This paper proposes 1) a complementary random masking strategy of RGB-T images and 2) self-distillation loss between clean and masked input modalities. We achieve state-of-the-art performance over three RGB-T semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2023-03-30T13:57:21Z)
• CIR-Net: Cross-modality Interaction and Refinement for RGB-D Salient Object Detection [144.66411561224507]
  We present a convolutional neural network (CNN) model, named CIR-Net, based on the novel cross-modality interaction and refinement. Our network outperforms the state-of-the-art saliency detectors both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-10-06T11:59:19Z)
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation [59.94819184452694]
  Depth information has proven to be a useful cue in the semantic segmentation of RGBD images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion. In this paper, we propose a unified and efficient Crossmodality Guided to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively.
  arXiv  Detail & Related papers  (2020-07-17T18:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.