CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection

• URL: http://arxiv.org/abs/2606.29136v1
• Date: Sun, 28 Jun 2026 00:49:05 GMT
• Title: CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection
• Abstract summary: Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range.<n>Previous multi-modal fusion techniques fail to handle the inherent heterogeneity between RGB frames and event streams.<n>We propose a Cross-Modal information inTeraction transFormer, which hierarchically integrates RGB and event information.
• Score: 17.487415888287
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range, compensating for the deficiencies of the conventional RGB frames. However, previous multi-modal fusion techniques typically fail to handle the inherent heterogeneity between RGB frames and event streams, thus easily leading to noise amplification or redundant feature integration during cross-modal fusion. In this paper, we propose a Cross-Modal information inTeraction transFormer, coined as CMTFormer, which hierarchically integrates RGB and event information to achieve efficient and stable multimodal collaboration. Specifically, we design a shallow-to-deep information interaction scheme. In the shallow stage, we present the Shallow Alignment Module (SAM) to achieve an efficient fusion of RGB and event low-level features, which mitigates attribute disparities and prevents noisy information. In the middle stage, we devise the Cross-modal Enhancement Module (CEM) that utilizes texture and edge information to produce mutually reinforced middle-level features. In the deep stage, we present the Learnable Deep Fusion Module (LDFM) which performs high-level information aggregation through learnable weights, thus enabling the network to adaptively fuse RGB and event clues. A Spatial Prior Module is further designed to utilize global spatial information to enhance localization accuracy. Extensive experiments are conducted on two prevalent event-based object detection benchmarks, i.e., DSEC-Detection and PKU-DAVIS-SOD. Our CMTFormer consistently surpasses the detection counterparts in both uni-modal and multi-modal settings, strongly demonstrating the effectiveness of our paradigm. Codes will be available upon publication.

Related papers

• HyPSAM: Hybrid Prompt-driven Segment Anything Model for RGB-Thermal Salient Object Detection [75.406055413928]
  We propose a novel prompt-driven segment anything model (HyPSAM) for RGB-T SOD.<n> DFNet employs dynamic convolution and multi-branch decoding to facilitate adaptive cross-modality interaction.<n>Experiments on three public datasets demonstrate that our method achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-09-23T07:32:11Z)
• Beyond conventional vision: RGB-event fusion for robust object detection in dynamic traffic scenarios [23.41380544271609]
  Dynamic range of conventional RGB cameras reduces global contrast and causes loss of high-frequency details.<n>We propose a motion cue fusion network (MCFNet) which achieves optimal cross-modal feature fusion under challenging lighting.<n>MCFNet significantly outperforms existing methods in various poor lighting and fast moving traffic scenarios.
  arXiv  Detail & Related papers  (2025-08-14T14:48:21Z)
• CIR-Net: Cross-modality Interaction and Refinement for RGB-D Salient Object Detection [144.66411561224507]
  We present a convolutional neural network (CNN) model, named CIR-Net, based on the novel cross-modality interaction and refinement. Our network outperforms the state-of-the-art saliency detectors both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-10-06T11:59:19Z)
• Dual Swin-Transformer based Mutual Interactive Network for RGB-D Salient Object Detection [67.33924278729903]
  In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the long-range dependencies in visual inputs. Comprehensive experiments on five standard RGB-D SOD benchmark datasets demonstrate the superiority of the proposed DTMINet method.
  arXiv  Detail & Related papers  (2022-06-07T08:35:41Z)
• Transformer-based Network for RGB-D Saliency Detection [82.6665619584628]
  Key to RGB-D saliency detection is to fully mine and fuse information at multiple scales across the two modalities. We show that transformer is a uniform operation which presents great efficacy in both feature fusion and feature enhancement. Our proposed network performs favorably against state-of-the-art RGB-D saliency detection methods.
  arXiv  Detail & Related papers  (2021-12-01T15:53:58Z)
• Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection [78.47767202232298]
  We propose a novel Cross-modality Discrepant Interaction Network (CDINet) for RGB-D SOD. Two components are designed to implement the effective cross-modality interaction. Our network outperforms $15$ state-of-the-art methods both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2021-08-04T11:24:42Z)
• RGB-D Salient Object Detection with Cross-Modality Modulation and Selection [126.4462739820643]
  We present an effective method to progressively integrate and refine the cross-modality complementarities for RGB-D salient object detection (SOD) The proposed network mainly solves two challenging issues: 1) how to effectively integrate the complementary information from RGB image and its corresponding depth map, and 2) how to adaptively select more saliency-related features.
  arXiv  Detail & Related papers  (2020-07-14T14:22:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.