AnyTrack: Unifying Visual Object Tracking with Any Modalities

• URL: http://arxiv.org/abs/2608.06773v1
• Date: Fri, 07 Aug 2026 03:47:34 GMT
• Title: AnyTrack: Unifying Visual Object Tracking with Any Modalities
• Abstract summary: We propose a unified framework called AnyTrack for object tracking with any modalities.<n> Modality-aware Interaction Module (MIM) to facilitate dynamic interaction across diverse modalities.<n> Context Understanding Module (CUM) to establish spatial correspondence between visual and target locations.
• Score: 71.788116599779
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual object tracking aims to continuously locate specific targets within sequential frames, evolving from single-modal methods to multi-modal ones. However, existing multi-modal trackers are typically designed for fixed modality combinations, requiring separate models for different inputs. This leads to a poor adaptability to missing or imperfect modalities, and limited generalization. To address these issues, we propose a novel unified framework called AnyTrack for object tracking with any modalities. Specifically, we design a Modality-aware Interaction Module (MIM) to facilitate dynamic interaction across diverse modalities. This module bridges modality discrepancies and aggregates temporal cues to maintain spatio-temporal consistency during cross-modal interaction. Furthermore, we introduce a Context Understanding Module (CUM) to establish spatial correspondence between visual features and target locations via global-local prompts. This module employs target-aware context modeling to enhance foreground-background discrimination for precise localization. Finally, to support the training and evaluation under diverse modalities, we extend existing multi-modal object tracking benchmarks by incorporating grayscale images, language descriptions, and audio clips. Extensive experiments with both complete and missing modality settings demonstrate that our AnyTrack achieves state-of-the-art performance, validating its effectiveness and flexibility. The source code is available at https://github.com/IdolLab/AnyTrack.

Related papers

• Unified Multimodal Visual Tracking with Dual Mixture-of-Experts [48.73939694967044]
  OneV2 is a unified multimodal tracking framework that enables end-to-end training for any modality.<n>OneV2 achieves state-of-the-art performance across five and RGB+X tracking tasks and 12 benchmarks.
  arXiv  Detail & Related papers  (2026-05-05T13:05:50Z)
• CAMELTrack: Context-Aware Multi-cue ExpLoitation for Online Multi-Object Tracking [68.24998698508344]
  We introduce CAMEL, a novel association module for Context-Aware Multi-Cue ExpLoitation.<n>Unlike end-to-end detection-by-tracking approaches, our method remains lightweight and fast to train while being able to leverage external off-the-shelf models.<n>Our proposed online tracking pipeline, CAMELTrack, achieves state-of-the-art performance on multiple tracking benchmarks.
  arXiv  Detail & Related papers  (2025-05-02T13:26:23Z)
• Adaptive Perception for Unified Visual Multi-modal Object Tracking [35.80808788962743]
  APTrack is a novel unified tracker designed for multi-modal adaptive perception.<n>Unlike previous methods, APTrack explores a unified representation through an equal modeling strategy.<n>Our tracker integrates an adaptive modality interaction module that efficiently bridges cross-modality interactions.
  arXiv  Detail & Related papers  (2025-02-10T15:50:26Z)
• SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection [79.58755811919366]
  This paper introduces a new task called Multi-Modal datasets and Multi-Task Object Detection (M2Det) for remote sensing.<n>It is designed to accurately detect horizontal or oriented objects from any sensor modality.<n>This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization.
  arXiv  Detail & Related papers  (2024-12-30T02:47:51Z)
• Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking [53.33637391723555]
  We propose a unified multimodal spatial-temporal tracking approach named STTrack.<n>In contrast to previous paradigms, we introduced a temporal state generator (TSG) that continuously generates a sequence of tokens containing multimodal temporal information.<n>These temporal information tokens are used to guide the localization of the target in the next time state, establish long-range contextual relationships between video frames, and capture the temporal trajectory of the target.
  arXiv  Detail & Related papers  (2024-12-20T09:10:17Z)
• Bi-directional Adapter for Multi-modal Tracking [67.01179868400229]
  We propose a novel multi-modal visual prompt tracking model based on a universal bi-directional adapter. We develop a simple but effective light feature adapter to transfer modality-specific information from one modality to another. Our model achieves superior tracking performance in comparison with both the full fine-tuning methods and the prompt learning-based methods.
  arXiv  Detail & Related papers  (2023-12-17T05:27:31Z)
• Discriminative Appearance Modeling with Multi-track Pooling for Real-time Multi-object Tracking [20.66906781151]
  In multi-object tracking, the tracker maintains in its memory the appearance and motion information for each object in the scene. Many approaches model each target in isolation and lack the ability to use all the targets in the scene to jointly update the memory. We propose a training strategy adapted to multi-track pooling which generates hard tracking episodes online.
  arXiv  Detail & Related papers  (2021-01-28T18:12:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.