Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

• URL: http://arxiv.org/abs/2605.03716v1
• Date: Tue, 05 May 2026 13:05:50 GMT
• Title: Unified Multimodal Visual Tracking with Dual Mixture-of-Experts
• Abstract summary: OneV2 is a unified multimodal tracking framework that enables end-to-end training for any modality.<n>OneV2 achieves state-of-the-art performance across five and RGB+X tracking tasks and 12 benchmarks.
• Score: 48.73939694967044
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal visual object tracking can be divided into to several kinds of tasks (e.g. RGB and RGB+X tracking), based on the input modality. Existing methods often train separate models for each modality or rely on pretrained models to adapt to new modalities, which limits efficiency, scalability, and usability. Thus, we introduce OneTrackerV2, a unified multi-modal tracking framework that enables end-to-end training for any modality. We propose Meta Merger to embed multi-modal information into a unified space, allowing flexible modality fusion and robustness. We further introduce Dual Mixture-of-Experts (DMoE): T-MoE models spatio-temporal relations for tracking, while M-MoE embeds multi-modal knowledge, disentangling cross-modal dependencies and reducing feature conflicts. With a shared architecture, unified parameters, and a single end-to-end training, OneTrackerV2 achieves state-of-the-art performance across five RGB and RGB+X tracking tasks and 12 benchmarks, while maintaining high inference efficiency. Notably, even after model compression, OneTrackerV2 retains strong performance. Moreover, OneTrackerV2 demonstrates remarkable robustness under modality-missing scenarios.

Related papers

• AnyTrack: Unifying Visual Object Tracking with Any Modalities [71.788116599779]
  We propose a unified framework called AnyTrack for object tracking with any modalities.<n> Modality-aware Interaction Module (MIM) to facilitate dynamic interaction across diverse modalities.<n> Context Understanding Module (CUM) to establish spatial correspondence between visual and target locations.
  arXiv  Detail & Related papers  (2026-08-07T03:47:34Z)
• Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking [16.123153889076104]
  We propose a novel Sparse-Dense Mixture of Experts Adapter (SDMoEA) framework for PEFT-based multi-modal tracking.<n>To overcome limitations of existing tracking methods in modeling high-order correlations during multi-level multi-modal fusion, we introduce a Gram-based Semantic Alignment Hypergraph Fusion (GSAHF) module.<n>The proposed method achieves superior performance compared with other PEFT approaches on several multi-modal tracking benchmarks.
  arXiv  Detail & Related papers  (2026-03-14T02:51:10Z)
• UBATrack: Spatio-Temporal State Space Model for General Multi-Modal Tracking [40.8191099453086]
  We introduce a novel multi-modal tracking framework based on a mamba-style state, termed UBATrack.<n>UBATrack comprises two simple yet effective work space: a S-temporal Mamba Adapter (MA) and a Dynamic Multi-modal Feature Mixer.<n>Experiments show that UBATrack outperforms state-of-the-art methods on RGB-T, RGB-D, and RGB-E tracking benchmarks.
  arXiv  Detail & Related papers  (2026-01-21T09:24:19Z)
• UniSOT: A Unified Framework for Multi-Modality Single Object Tracking [60.21741689231572]
  We present a unified tracker, UniSOT, for different combinations of three reference modalities and four video modalities with uniform parameters.<n>UniSOT shows superior performance against modality-specific counterparts.<n> Notably, UniSOT outperforms previous counterparts by over 3.0% AUC on TNL2K across all three reference modalities and outperforms Un-Track by over 2.0% main metric across all three RGB+X video modalities.
  arXiv  Detail & Related papers  (2025-11-03T10:23:53Z)
• Adaptive Perception for Unified Visual Multi-modal Object Tracking [35.80808788962743]
  APTrack is a novel unified tracker designed for multi-modal adaptive perception.<n>Unlike previous methods, APTrack explores a unified representation through an equal modeling strategy.<n>Our tracker integrates an adaptive modality interaction module that efficiently bridges cross-modality interactions.
  arXiv  Detail & Related papers  (2025-02-10T15:50:26Z)
• SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection [79.58755811919366]
  This paper introduces a new task called Multi-Modal datasets and Multi-Task Object Detection (M2Det) for remote sensing.<n>It is designed to accurately detect horizontal or oriented objects from any sensor modality.<n>This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization.
  arXiv  Detail & Related papers  (2024-12-30T02:47:51Z)
• Bi-directional Adapter for Multi-modal Tracking [67.01179868400229]
  We propose a novel multi-modal visual prompt tracking model based on a universal bi-directional adapter. We develop a simple but effective light feature adapter to transfer modality-specific information from one modality to another. Our model achieves superior tracking performance in comparison with both the full fine-tuning methods and the prompt learning-based methods.
  arXiv  Detail & Related papers  (2023-12-17T05:27:31Z)
• Prompting for Multi-Modal Tracking [70.0522146292258]
  We propose a novel multi-modal prompt tracker (ProTrack) for multi-modal tracking. ProTrack can transfer the multi-modal inputs to a single modality by the prompt paradigm. Our ProTrack can achieve high-performance multi-modal tracking by only altering the inputs, even without any extra training on multi-modal data.
  arXiv  Detail & Related papers  (2022-07-29T09:35:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.