Bridging the RGB-IR Gap: Consensus and Discrepancy Modeling for Text-Guided Multispectral Detection

• URL: http://arxiv.org/abs/2604.11234v1
• Date: Mon, 13 Apr 2026 09:41:01 GMT
• Title: Bridging the RGB-IR Gap: Consensus and Discrepancy Modeling for Text-Guided Multispectral Detection
• Abstract summary: We propose a semantic bridge fusion framework with bi-support modeling for multispectral object detection.<n>Specifically, text is used as a shared semantic bridge to align RGB and IR responses under a unified category condition.<n>We formulate RGB-IR interaction evidence into the regular consensus support and the complementary discrepancy support.
• Score: 45.62297680190076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-guided multispectral object detection uses text semantics to guide semantic-aware cross-modal interaction between RGB and IR for more robust perception. However, notable limitations remain: (1) existing methods often use text only as an auxiliary semantic enhancement signal, without exploiting its guiding role to bridge the inherent granularity asymmetry between RGB and IR; and (2) conventional data-driven attention-based fusion tends to emphasize stable consensus while overlooking potentially valuable cross-modal discrepancies. To address these issues, we propose a semantic bridge fusion framework with bi-support modeling for multispectral object detection. Specifically, text is used as a shared semantic bridge to align RGB and IR responses under a unified category condition, while the recalibrated thermal semantic prior is projected onto the RGB branch for semantic-level mapping fusion. We further formulate RGB-IR interaction evidence into the regular consensus support and the complementary discrepancy support that contains potentially discriminative cues, and introduce them into fusion via dynamic recalibration as a structured inductive bias. In addition, we design a bidirectional semantic alignment module for closed-loop vision-text guidance enhancement. Extensive experiments demonstrate the effectiveness of the proposed fusion framework and its superior detection performance on multispectral benchmarks. Code is available at https://github.com/zhenwang5372/Bridging-RGB-IR-Gap.

Related papers

• Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration [17.74037850404483]
  We propose an end-to-end visible-infrared oriented object detector tailored for severely cross-modal geometric discrepancies.<n>The proposed JFRDet achieves 69.7% $mathrmmAP_50$ on DroneVehicle Misaligned (DVMA), which represents state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-08-11T09:02:37Z)
• Beyond Symmetric Fusion: Exploiting Task-Dependent Modality Strengths for RGB-Event Small Object Detection [12.262266865349254]
  State-of-the-art RGB-Event detectors improve the detection of small, fast-moving objects by combining complementary features from RGB and Event data.<n>Event data are substantially more effective for class-agnostic localization, whereas RGB data provide stronger category evidence within localized target regions.<n>Motivated by this task-dependent asymmetry, we propose an Asymmetric Event-RGB Object Detection Transformer (AERODet)
  arXiv  Detail & Related papers  (2026-08-02T15:10:54Z)
• Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing [55.13802890769086]
  We present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing.<n>To achieve profound modal synergy, Evita embeds a suite of intrinsic co-learning modules directly into every encoder layer.<n>Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time multimodal perception.
  arXiv  Detail & Related papers  (2026-07-10T06:52:09Z)
• Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentation [18.450662919776757]
  We propose a novel Edge-awareness Semantic Concordance framework to unify the multi-modality heterogeneous features with latent edge cues.<n>Our method outperforms the state-of-the-art by a 2.55% mIoU on our proposed DERS-XS.
  arXiv  Detail & Related papers  (2025-11-11T14:00:27Z)
• RGBX-DiffusionDet: A Framework for Multi-Modal RGB-X Object Detection Using DiffusionDet [0.0]
  RGBX-DiffusionDet is an object detection framework extending the DiffusionDet model.<n>It fuses the heterogeneous 2D data (X) with RGB imagery via an adaptive multimodal encoder.
  arXiv  Detail & Related papers  (2025-05-05T11:39:51Z)
• Bringing RGB and IR Together: Hierarchical Multi-Modal Enhancement for Robust Transmission Line Detection [67.02804741856512]
  We propose a novel Hierarchical Multi-Modal Enhancement Network (HMMEN) that integrates RGB and IR data for robust and accurate TL detection.<n>Our method introduces two key components: (1) a Mutual Multi-Modal Enhanced Block (MMEB), which fuses and enhances hierarchical RGB and IR feature maps in a coarse-to-fine manner, and (2) a Feature Alignment Block (FAB) that corrects misalignments between decoder outputs and IR feature maps by leveraging deformable convolutions.
  arXiv  Detail & Related papers  (2025-01-25T06:21:06Z)
• DEYOLO: Dual-Feature-Enhancement YOLO for Cross-Modality Object Detection [5.946464547429392]
  Object detection in poor-illumination environments is a challenging task as objects are usually not clearly visible in RGB images.<n>We propose a dual-enhancement-based cross-modality object detection network DEYOLO.<n>Our approach outperforms SOTA object detection algorithms by a clear margin.
  arXiv  Detail & Related papers  (2024-12-06T10:39:11Z)
• Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation [19.41334573257174]
  Traditional methods mostly use RGB images which are heavily affected by lighting conditions, eg, darkness. Recent studies show thermal images are robust to the night scenario as a compensating modality for segmentation. This work proposes a Residual Spatial Fusion Network (RSFNet) for RGB-T semantic segmentation.
  arXiv  Detail & Related papers  (2023-06-17T14:28:08Z)
• CIR-Net: Cross-modality Interaction and Refinement for RGB-D Salient Object Detection [144.66411561224507]
  We present a convolutional neural network (CNN) model, named CIR-Net, based on the novel cross-modality interaction and refinement. Our network outperforms the state-of-the-art saliency detectors both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-10-06T11:59:19Z)
• Dual Swin-Transformer based Mutual Interactive Network for RGB-D Salient Object Detection [67.33924278729903]
  In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the long-range dependencies in visual inputs. Comprehensive experiments on five standard RGB-D SOD benchmark datasets demonstrate the superiority of the proposed DTMINet method.
  arXiv  Detail & Related papers  (2022-06-07T08:35:41Z)
• Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection [78.47767202232298]
  We propose a novel Cross-modality Discrepant Interaction Network (CDINet) for RGB-D SOD. Two components are designed to implement the effective cross-modality interaction. Our network outperforms $15$ state-of-the-art methods both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2021-08-04T11:24:42Z)
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation [59.94819184452694]
  Depth information has proven to be a useful cue in the semantic segmentation of RGBD images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion. In this paper, we propose a unified and efficient Crossmodality Guided to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively.
  arXiv  Detail & Related papers  (2020-07-17T18:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.