Robust RGB-T Tracking via Learnable Visual Fourier Prompt Fine-tuning and Modality Fusion Prompt Generation

• URL: http://arxiv.org/abs/2509.19733v1
• Date: Wed, 24 Sep 2025 03:26:25 GMT
• Title: Robust RGB-T Tracking via Learnable Visual Fourier Prompt Fine-tuning and Modality Fusion Prompt Generation
• Authors: Hongtao Yang, Bineng Zhong, Qihua Liang, Zhiruo Zhu, Yaozong Zheng, Ning Li,
• Abstract summary: We propose an efficient Visual Fourier Prompt Tracking method to learn modality-related prompts via Fast Fourier Transform (FFT)<n>Our method consists of symmetric feature extraction encoder with shared parameters, visual fourier prompts, and Modality Fusion Prompt Generator.<n>Experiments conducted on three popular RGB-T tracking benchmarks show that our method demonstrates outstanding performance.
• Score: 32.437441219889
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, visual prompt tuning is introduced to RGB-Thermal (RGB-T) tracking as a parameter-efficient finetuning (PEFT) method. However, these PEFT-based RGB-T tracking methods typically rely solely on spatial domain information as prompts for feature extraction. As a result, they often fail to achieve optimal performance by overlooking the crucial role of frequency-domain information in prompt learning. To address this issue, we propose an efficient Visual Fourier Prompt Tracking (named VFPTrack) method to learn modality-related prompts via Fast Fourier Transform (FFT). Our method consists of symmetric feature extraction encoder with shared parameters, visual fourier prompts, and Modality Fusion Prompt Generator that generates bidirectional interaction prompts through multi-modal feature fusion. Specifically, we first use a frozen feature extraction encoder to extract RGB and thermal infrared (TIR) modality features. Then, we combine the visual prompts in the spatial domain with the frequency domain prompts obtained from the FFT, which allows for the full extraction and understanding of modality features from different domain information. Finally, unlike previous fusion methods, the modality fusion prompt generation module we use combines features from different modalities to generate a fused modality prompt. This modality prompt is interacted with each individual modality to fully enable feature interaction across different modalities. Extensive experiments conducted on three popular RGB-T tracking benchmarks show that our method demonstrates outstanding performance.

Related papers

• Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm [103.36490810025752]
  Existing multi-modal object tracking approaches primarily focus on dual-modal paradigms, such as RGB-Depth or RGB-Thermal.<n>This work introduces a novel multi-modal tracking task that leverages three complementary modalities, including visible RGB, Depth (D), and Thermal Infrared (TIR)<n>We propose a novel multi-modal tracker, dubbed RDTTrack, which integrates tri-modal information for robust tracking by leveraging a pretrained RGB-only tracking model.
  arXiv  Detail & Related papers  (2025-09-29T13:05:15Z)
• HyPSAM: Hybrid Prompt-driven Segment Anything Model for RGB-Thermal Salient Object Detection [75.406055413928]
  We propose a novel prompt-driven segment anything model (HyPSAM) for RGB-T SOD.<n> DFNet employs dynamic convolution and multi-branch decoding to facilitate adaptive cross-modality interaction.<n>Experiments on three public datasets demonstrate that our method achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-09-23T07:32:11Z)
• Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking [74.15663758681849]
  We present Learning Frequency and Memory-Aware Prompts, a dual-adapter framework that injects lightweight prompts into a frozen RGB tracker.<n>A frequency-guided visual adapter adaptively transfers complementary cues across modalities.<n>A multilevel memory adapter with short, long, and permanent memory stores, updates, and retrieves reliable temporal context.
  arXiv  Detail & Related papers  (2025-06-30T15:38:26Z)
• Diff-MM: Exploring Pre-trained Text-to-Image Generation Model for Unified Multi-modal Object Tracking [45.341224888996514]
  Multi-modal object tracking integrates auxiliary modalities such as depth, thermal infrared, event flow, and language.<n>Existing methods typically start from an RGB-based tracker and learn to understand auxiliary modalities only from training data.<n>This work proposes a unified multi-modal tracker Diff-MM by exploiting the multi-modal understanding capability of the pre-trained text-to-image generation model.
  arXiv  Detail & Related papers  (2025-05-19T01:42:13Z)
• Deep Fourier-embedded Network for RGB and Thermal Salient Object Detection [15.470610918037243]
  We propose a purely Fourier Transform-based model, namely Deep Fourier-embedded Network (FreqSal) for accurate RGB-T SOD.<n>Specifically, we leverage the efficiency of Fast Fourier Transform with linear complexity to design three key components.<n>Experiments on ten bimodal SOD benchmark datasets demonstrate that FreqSal outperforms twenty-nine existing state-of-the-art bimodal SOD models.
  arXiv  Detail & Related papers  (2024-11-27T14:55:16Z)
• Coordinate-Aware Thermal Infrared Tracking Via Natural Language Modeling [16.873697155916997]
  NLMTrack is a coordinate-aware thermal infrared tracking model. NLMTrack applies an encoder that unifies feature extraction and feature fusion. Experiments show that NLMTrack achieves state-of-the-art performance on multiple benchmarks.
  arXiv  Detail & Related papers  (2024-07-11T08:06:31Z)
• XTrack: Multimodal Training Boosts RGB-X Video Object Trackers [88.72203975896558]
  It is crucial to ensure that knowledge gained from multimodal sensing is effectively shared.<n>Similar samples across different modalities have more knowledge to share than otherwise.<n>We propose a method for RGB-X tracker during inference, with an average +3% precision improvement over the current SOTA.
  arXiv  Detail & Related papers  (2024-05-28T03:00:58Z)
• Transformer-based RGB-T Tracking with Channel and Spatial Feature Fusion [4.963745612929956]
  The main problem in RGB-T tracking is the correct and optimal merging of the cross-modal features of visible and thermal images.<n>CSTNet aims to achieve a direct fusion of cross-modal channels and spatial features in RGB-T tracking.<n>CSTNet and CSTNet-small achieve real-time speeds of 21 fps and 33 fps on the Nvidia Jetson Xavier.
  arXiv  Detail & Related papers  (2024-05-06T05:58:49Z)
• Bi-directional Adapter for Multi-modal Tracking [67.01179868400229]
  We propose a novel multi-modal visual prompt tracking model based on a universal bi-directional adapter. We develop a simple but effective light feature adapter to transfer modality-specific information from one modality to another. Our model achieves superior tracking performance in comparison with both the full fine-tuning methods and the prompt learning-based methods.
  arXiv  Detail & Related papers  (2023-12-17T05:27:31Z)
• RGB-T Tracking Based on Mixed Attention [5.151994214135177]
  RGB-T tracking involves the use of images from both visible and thermal modalities. An RGB-T tracker based on mixed attention mechanism to achieve complementary fusion of modalities is proposed in this paper.
  arXiv  Detail & Related papers  (2023-04-09T15:59:41Z)
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion [138.40422469153145]
  We propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. We show that CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion.
  arXiv  Detail & Related papers  (2022-11-26T02:40:28Z)
• Dual Swin-Transformer based Mutual Interactive Network for RGB-D Salient Object Detection [67.33924278729903]
  In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the long-range dependencies in visual inputs. Comprehensive experiments on five standard RGB-D SOD benchmark datasets demonstrate the superiority of the proposed DTMINet method.
  arXiv  Detail & Related papers  (2022-06-07T08:35:41Z)
• Temporal Aggregation for Adaptive RGBT Tracking [14.00078027541162]
  We propose an RGBT tracker which takes clues into account for robust appearance model learning. Unlike most existing RGBT trackers that implement object tracking tasks with only spatial information included, temporal information is further considered in this method.
  arXiv  Detail & Related papers  (2022-01-22T02:31:56Z)
• Transformer-based Network for RGB-D Saliency Detection [82.6665619584628]
  Key to RGB-D saliency detection is to fully mine and fuse information at multiple scales across the two modalities. We show that transformer is a uniform operation which presents great efficacy in both feature fusion and feature enhancement. Our proposed network performs favorably against state-of-the-art RGB-D saliency detection methods.
  arXiv  Detail & Related papers  (2021-12-01T15:53:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.