Dual Sparse Aggregation Transformer for Multispectral Object Detection

• URL: http://arxiv.org/abs/2606.31015v1
• Date: Tue, 30 Jun 2026 01:05:24 GMT
• Title: Dual Sparse Aggregation Transformer for Multispectral Object Detection
• Abstract summary: We propose a novel Dual Sparse Aggregation Transformer (DSAFormer) for multispectral object detection.<n>Specifically, the DSAFormer is designed to exploit and boost cross-modal complementary information.<n>Our proposed DSAFormer achieves better detection performance against state-of-the-art methods on four public datasets.
• Score: 34.20084627163684
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based approaches have obtained excellent performance in multispectral object detection tasks due to their ability to model long-range dependencies and capture complementary information. However, previous transformer-based multispectral detection methods tend to use all available tokens for similarity calculation, which results in redundant information interaction from irrelevant areas, leading to degraded detection performance. To overcome this challenge, we propose a novel Dual Sparse Aggregation Transformer (DSAFormer) for multispectral object detection, which consists of a Dual Sparse Transformer (DSFormer) and a Learnable Addition Fusion Block (LAFB). Specifically, the DSFormer is designed to exploit and boost cross-modal complementary information, thereby improving detection performance. It incorporates three key components: A Spatial Sparse Multi-Head Cross-Attention (SSMHCA) mechanism selectively captures cross-modal relationships at the spatial level by reserving only the high query-key similarity scores, eliminating irrelevant interactions. A Channel Sparse Multi-Head Cross-Attention (CSMHCA) mechanism performs similar sparse calculations at the channel level to enhance feature representation and filter out low matching query-key. A Multi-Scale Feature Refinement Layer (MSFRL) is developed to aggregate hierarchical features and suppress redundant information. To effectively fuse multimodal features, the LAFB is introduced to aggregate intramodal and intermodal feature information by feature reweighting. Extensive experimental results have demonstrated that our proposed DSAFormer achieves better detection performance against state-of-the-art methods on four public datasets, including the MFAD, FLIR, M$^3$FD, and LLVIP. The source code of our DSAFormer will be released at https://github.com/WenCongWu/DSAFormer.

Related papers

• S$^3$AM: A Single-Stream SAM with Reliability-Calibrated Frequency Adapter for Multi-modal Salient Object Detection [74.15663758681849]
  Single-stream framework integrates reliability-calibrated frequency adaptation into the adopted SAM backbone for MSOD.<n>Experiments on RGB-D, RGB-T, and RGB-NIR salient object detection benchmarks validate that the proposed framework achieves competitive performance.
  arXiv  Detail & Related papers  (2026-08-18T08:05:00Z)
• Source-Free Object Detection with Detection Transformer [59.33653163035064]
  Source-Free Object Detection (SFOD) enables knowledge transfer from a source domain to an unsupervised target domain for object detection without access to source data.<n>Most existing SFOD approaches are either confined to conventional object detection (OD) models like Faster R-CNN or designed as general solutions without tailored adaptations for novel OD architectures, especially Detection Transformer (DETR)<n>In this paper, we introduce Feature Reweighting ANd Contrastive Learning NetworK (FRANCK), a novel SFOD framework specifically designed to perform query-centric feature enhancement for DETRs.
  arXiv  Detail & Related papers  (2025-10-13T07:35:04Z)
• Cross Spatial Temporal Fusion Attention for Remote Sensing Object Detection via Image Feature Matching [15.57849268814515]
  We propose a mechanism that enhances feature representation by integrating scale-invariant keypoints detected independently in both reference and query images.<n>Our approach improves feature matching in two ways: First, by creating correspondence maps that leverage information from multiple image regions simultaneously, and second, by reformulating the similarity matching process as a classification task.<n>To demonstrate the practical utility of improved feature matching, we evaluate CSTF on object detection tasks using the HRSC2016 and DOTA benchmark datasets.
  arXiv  Detail & Related papers  (2025-07-25T09:52:06Z)
• Dual-Perspective United Transformer for Object Segmentation in Optical Remote Sensing Images [38.942152581251165]
  We propose a novel Dual-Perspective United Transformer (DPU-Former) with a unique structure designed to simultaneously integrate long-range dependencies and spatial details.<n>In particular, we design the global-local mixed attention, which captures diverse information through two perspectives.<n>We present a gated linear feed-forward network to increase the expressive ability.
  arXiv  Detail & Related papers  (2025-06-27T02:40:48Z)
• Prototype-Based Information Compensation Network for Multi-Source Remote Sensing Data Classification [56.065032039986725]
  Multi-source remote sensing data joint classification aims to provide accuracy and reliability of land cover classification.<n>Existing methods confront two challenges: inter-frequency multi-source feature coupling and inconsistency of complementary information exploration.<n>We present a Prototype-based Information Compensation Network (PICNet) for land cover classification based on HSI and SAR/LiDAR data.
  arXiv  Detail & Related papers  (2025-05-06T22:30:23Z)
• SeaDATE: Remedy Dual-Attention Transformer with Semantic Alignment via Contrast Learning for Multimodal Object Detection [18.090706979440334]
  Multimodal object detection leverages diverse modal information to enhance the accuracy and robustness of detectors. Current methods merely stack Transformer-guided fusion techniques without exploring their capability to extract features at various depth layers of network. In this paper, we introduce an accurate and efficient object detection method named SeaDATE.
  arXiv  Detail & Related papers  (2024-10-15T07:26:39Z)
• PoIFusion: Multi-Modal 3D Object Detection via Fusion at Points of Interest [65.48057241587398]
  PoIFusion is a framework to fuse information of RGB images and LiDAR point clouds at the points of interest (PoIs) Our approach maintains the view of each modality and obtains multi-modal features by computation-friendly projection and computation. We conducted extensive experiments on nuScenes and Argoverse2 datasets to evaluate our approach.
  arXiv  Detail & Related papers  (2024-03-14T09:28:12Z)
• Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Images [1.662438436885552]
  Multi-modal fusion has been determined to enhance the accuracy by fusing data from multiple modalities. We propose a novel multi-modal fusion strategy for mapping relationships between different channels at the early stage. By addressing fusion in the early stage, as opposed to mid or late-stage methods, our method achieves competitive and even superior performance compared to existing techniques.
  arXiv  Detail & Related papers  (2023-10-21T00:56:11Z)
• ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection [25.66305300362193]
  A novel feature fusion framework of dual cross-attention transformers is proposed to model global feature interaction. This framework enhances the discriminability of object features through the query-guided cross-attention mechanism. The proposed method achieves superior performance and faster inference, making it suitable for various practical scenarios.
  arXiv  Detail & Related papers  (2023-08-15T00:02:10Z)
• Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection [54.041049052843604]
  We present STEMD, a novel end-to-end framework that enhances the DETR-like paradigm for multi-frame 3D object detection. First, to model the inter-object spatial interaction and complex temporal dependencies, we introduce the spatial-temporal graph attention network. Finally, it poses a challenge for the network to distinguish between the positive query and other highly similar queries that are not the best match.
  arXiv  Detail & Related papers  (2023-07-01T13:53:14Z)
• Transformer-based Network for RGB-D Saliency Detection [82.6665619584628]
  Key to RGB-D saliency detection is to fully mine and fuse information at multiple scales across the two modalities. We show that transformer is a uniform operation which presents great efficacy in both feature fusion and feature enhancement. Our proposed network performs favorably against state-of-the-art RGB-D saliency detection methods.
  arXiv  Detail & Related papers  (2021-12-01T15:53:58Z)
• AFD-Net: Adaptive Fully-Dual Network for Few-Shot Object Detection [8.39479809973967]
  Few-shot object detection (FSOD) aims at learning a detector that can fast adapt to previously unseen objects with scarce examples. Existing methods solve this problem by performing subtasks of classification and localization utilizing a shared component. We present that a general few-shot detector should consider the explicit decomposition of two subtasks, as well as leveraging information from both of them to enhance feature representations.
  arXiv  Detail & Related papers  (2020-11-30T10:21:32Z)
• Multi-scale Interactive Network for Salient Object Detection [91.43066633305662]
  We propose the aggregate interaction modules to integrate the features from adjacent levels. To obtain more efficient multi-scale features, the self-interaction modules are embedded in each decoder unit. Experimental results on five benchmark datasets demonstrate that the proposed method without any post-processing performs favorably against 23 state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-07-17T15:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.