AFter: Attention-based Fusion Router for RGBT Tracking

• URL: http://arxiv.org/abs/2405.02717v1
• Date: Sat, 4 May 2024 17:24:09 GMT
• Title: AFter: Attention-based Fusion Router for RGBT Tracking
• Authors: Andong Lu, Wanyu Wang, Chenglong Li, Jin Tang, Bin Luo,
• Abstract summary: Existing RGBT tracking methods widely adopt fixed fusion structures to integrate multi-modal feature. We develop a novel emphAttention-based emphFusion rouemphter called AFter, which optimize the fusion structure to adapt to challenging scenarios. In particular, we design a fusion structure space based on the hierarchical attention network, each attention-based fusion unit corresponding to a fusion operation and a combination of these attention units corresponding to a fusion structure.
• Score: 22.449878625622844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal feature fusion as a core investigative component of RGBT tracking emerges numerous fusion studies in recent years. However, existing RGBT tracking methods widely adopt fixed fusion structures to integrate multi-modal feature, which are hard to handle various challenges in dynamic scenarios. To address this problem, this work presents a novel \emph{A}ttention-based \emph{F}usion rou\emph{ter} called AFter, which optimizes the fusion structure to adapt to the dynamic challenging scenarios, for robust RGBT tracking. In particular, we design a fusion structure space based on the hierarchical attention network, each attention-based fusion unit corresponding to a fusion operation and a combination of these attention units corresponding to a fusion structure. Through optimizing the combination of attention-based fusion units, we can dynamically select the fusion structure to adapt to various challenging scenarios. Unlike complex search of different structures in neural architecture search algorithms, we develop a dynamic routing algorithm, which equips each attention-based fusion unit with a router, to predict the combination weights for efficient optimization of the fusion structure. Extensive experiments on five mainstream RGBT tracking datasets demonstrate the superior performance of the proposed AFter against state-of-the-art RGBT trackers. We release the code in https://github.com/Alexadlu/AFter.

Related papers

• Dynamic Disentangled Fusion Network for RGBT Tracking [26.99387895981277]
  We propose a novel Dynamic Disentangled Fusion Network called DDFNet, which disentangles the fusion process into several dynamic fusion models. In particular, we design six attribute-based fusion models to integrate RGB and thermal features under six challenging scenarios. Experimental results on benchmark datasets demonstrate the effectiveness of our DDFNet against other state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-12-11T15:03:27Z)
• RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba [22.449878625622844]
  This paper presents a novel All-layer multimodal Interaction Network, named AINet, for robust RGBT tracking. We show that AINet achieves leading performance against existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-16T16:22:34Z)
• DAMSDet: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion [82.2425759608975]
  Infrared-visible object detection aims to achieve robust even full-day object detection by fusing the complementary information of infrared and visible images. We propose a Dynamic Adaptive Multispectral Detection Transformer (DAMSDet) to address these two challenges. Experiments on four public datasets demonstrate significant improvements compared to other state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-03-01T07:03:27Z)
• ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection [25.66305300362193]
  A novel feature fusion framework of dual cross-attention transformers is proposed to model global feature interaction. This framework enhances the discriminability of object features through the query-guided cross-attention mechanism. The proposed method achieves superior performance and faster inference, making it suitable for various practical scenarios.
  arXiv  Detail & Related papers  (2023-08-15T00:02:10Z)
• RGBT Tracking via Progressive Fusion Transformer with Dynamically Guided Learning [37.067605349559]
  We propose a novel Progressive Fusion Transformer called ProFormer. It integrates single-modality information into the multimodal representation for robust RGBT tracking. ProFormer sets a new state-of-the-art performance on RGBT210, RGBT234, LasHeR, and VTUAV datasets.
  arXiv  Detail & Related papers  (2023-03-26T16:55:58Z)
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion [138.40422469153145]
  We propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. We show that CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion.
  arXiv  Detail & Related papers  (2022-11-26T02:40:28Z)
• Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion [112.27103169303184]
  Multimodal Knowledge Graphs (MKGs) organize visual-text factual knowledge. MKGformer can obtain SOTA performance on four datasets of multimodal link prediction, multimodal RE, and multimodal NER.
  arXiv  Detail & Related papers  (2022-05-04T23:40:04Z)
• High-Performance Transformer Tracking [74.07751002861802]
  We present a Transformer tracking (named TransT) method based on the Siamese-like feature extraction backbone, the designed attention-based fusion mechanism, and the classification and regression head. Experiments show that our TransT and TransT-M methods achieve promising results on seven popular datasets.
  arXiv  Detail & Related papers  (2022-03-25T09:33:29Z)
• Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking [1.015785232738621]
  We address the problem of multi-modal object tracking in video. We explore various options of fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities.
  arXiv  Detail & Related papers  (2022-01-21T12:37:43Z)
• Cross-Modality Fusion Transformer for Multispectral Object Detection [0.0]
  Multispectral image pairs can provide the combined information, making object detection applications more reliable and robust. We present a simple yet effective cross-modality feature fusion approach, named Cross-Modality Fusion Transformer (CFT) in this paper.
  arXiv  Detail & Related papers  (2021-10-30T15:34:12Z)
• Transformer Tracking [76.96796612225295]
  Correlation acts as a critical role in the tracking field, especially in popular Siamese-based trackers. This work presents a novel attention-based feature fusion network, which effectively combines the template and search region features solely using attention. Experiments show that our TransT achieves very promising results on six challenging datasets.
  arXiv  Detail & Related papers  (2021-03-29T09:06:55Z)
• Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation [144.50154657257605]
  We propose an efficient framework to simultaneously search for all main components including backbone, segmentation branches, and feature fusion module. Our searched architecture, namely Auto-Panoptic, achieves the new state-of-the-art on the challenging COCO and ADE20K benchmarks.
  arXiv  Detail & Related papers  (2020-10-30T08:34:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.