QTrack: Query-Driven Reasoning for Multi-modal MOT

• URL: http://arxiv.org/abs/2603.13759v1
• Date: Sat, 14 Mar 2026 05:11:36 GMT
• Title: QTrack: Query-Driven Reasoning for Multi-modal MOT
• Abstract summary: Multi-object tracking (MOT) has traditionally focused on estimating of all objects in a video, without selectively reasoning about user-specified targets.<n>We introduce a query-driven tracking paradigm that formulates tracking as atemporal reasoning problem conditioned on natural language queries.<n>We present QTrack, an end-to-end vision-language model that integrates multimodal reasoning with tracking-oriented localization.
• Score: 1.9981885081131854
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-object tracking (MOT) has traditionally focused on estimating trajectories of all objects in a video, without selectively reasoning about user-specified targets under semantic instructions. In this work, we introduce a query-driven tracking paradigm that formulates tracking as a spatiotemporal reasoning problem conditioned on natural language queries. Given a reference frame, a video sequence, and a textual query, the goal is to localize and track only the target(s) specified in the query while maintaining temporal coherence and identity consistency. To support this setting, we construct RMOT26, a large-scale benchmark with grounded queries and sequence-level splits to prevent identity leakage and enable robust evaluation of generalization. We further present QTrack, an end-to-end vision-language model that integrates multimodal reasoning with tracking-oriented localization. Additionally, we introduce a Temporal Perception-Aware Policy Optimization strategy with structured rewards to encourage motion-aware reasoning. Extensive experiments demonstrate the effectiveness of our approach for reasoning-centric, language-guided tracking. Code and data are available at https://github.com/gaash-lab/QTrack

Related papers

• ReferTrack: Referring Then Tracking for Embodied Visual Tracking [13.368122578638847]
  Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language.<n>We introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera.<n>Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision.
  arXiv  Detail & Related papers  (2026-07-22T12:05:13Z)
• Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking [61.91472604294714]
  We propose a novel language dependency parsing mechanism to distill tracking principal components.<n>We exploit the powerful cross-modal understanding capability of the pre-trained vision-language model Qwen-VL.<n>Our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks.
  arXiv  Detail & Related papers  (2026-06-28T12:12:18Z)
• Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation [60.96010213186819]
  We introduce Interactive Tracking, a new paradigm that allows users to guide the tracker at any time using natural language commands.<n>We present InteractTrack, the first large-scale benchmark for interactive tracking, containing 150 videos with dense bounding box annotations and timestamped language instructions.<n>Third, we introduce Interactive Memory-Augmented Tracking (IMAT), a new baseline that employs a dynamic memory mechanism to learn from user feedback and update tracking behavior accordingly.
  arXiv  Detail & Related papers  (2026-04-02T12:33:27Z)
• ReasoningTrack: Chain-of-Thought Reasoning for Long-term Vision-Language Tracking [18.491855733401742]
  This paper proposes a novel reasoning-based vision-language tracking framework, named ReasoningTrack, based on a pre-trained vision-language model Qwen2.5-VL.<n>We embed the updated language descriptions and feed them into a unified tracking backbone network together with vision features.<n>In addition, we propose a large-scale long-term vision-language tracking benchmark dataset, termed TNLLT, which contains 200 video sequences.
  arXiv  Detail & Related papers  (2025-08-07T10:02:07Z)
• ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Tracking [23.76697700853566]
  We propose a new task, called Reasoning-based Multi-Object Tracking (ReaMOT)<n>ReaMOT is a more challenging task that requires accurate reasoning about objects that match the language instruction with reasoning characteristic and tracking the objects' trajectories.<n>We construct ReaMOT Challenge, a reasoning-based multi-object tracking benchmark built upon 12 datasets.
  arXiv  Detail & Related papers  (2025-05-26T17:55:19Z)
• Less is More: Token Context-aware Learning for Object Tracking [20.222950380244377]
  LMTrack is a token context-aware tracking pipeline.<n>It automatically learns high-quality reference tokens for efficient visual tracking.<n>It achieves state-of-the-art results on tracking benchmarks such as GOT-10K, TrackingNet, and LaSOT.
  arXiv  Detail & Related papers  (2025-01-01T07:05:31Z)
• SynCL: A Synergistic Training Strategy with Instance-Aware Contrastive Learning for End-to-End Multi-Camera 3D Tracking [34.90147791481045]
  SynCL is a novel plug-and-play synergistic training strategy designed to co-facilitate multi-task learning for detection and tracking.<n>We propose a Task-specific Hybrid Matching module for a weight-shared cross-attention-based decoder.<n>We also introduce Instance-aware Contrastive Learning to break through the barrier of self-centric attention for track queries.
  arXiv  Detail & Related papers  (2024-11-11T08:18:49Z)
• RTracker: Recoverable Tracking via PN Tree Structured Memory [71.05904715104411]
  We propose a recoverable tracking framework, RTracker, that uses a tree-structured memory to dynamically associate a tracker and a detector to enable self-recovery. Specifically, we propose a Positive-Negative Tree-structured memory to chronologically store and maintain positive and negative target samples. Our core idea is to use the support samples of positive and negative target categories to establish a relative distance-based criterion for a reliable assessment of target loss.
  arXiv  Detail & Related papers  (2024-03-28T08:54:40Z)
• Tracking with Human-Intent Reasoning [64.69229729784008]
  This work proposes a new tracking task -- Instruction Tracking. It involves providing implicit tracking instructions that require the trackers to perform tracking automatically in video frames. TrackGPT is capable of performing complex reasoning-based tracking.
  arXiv  Detail & Related papers  (2023-12-29T03:22:18Z)
• Towards Unified Token Learning for Vision-Language Tracking [65.96561538356315]
  We present a vision-language (VL) tracking pipeline, termed textbfMMTrack, which casts VL tracking as a token generation task. Our proposed framework serializes language description and bounding box into a sequence of discrete tokens. In this new design paradigm, all token queries are required to perceive the desired target and directly predict spatial coordinates of the target.
  arXiv  Detail & Related papers  (2023-08-27T13:17:34Z)
• Tracking Objects and Activities with Attention for Temporal Sentence Grounding [51.416914256782505]
  Temporal sentence (TSG) aims to localize the temporal segment which is semantically aligned with a natural language query in an untrimmed segment. We propose a novel Temporal Sentence Tracking Network (TSTNet), which contains (A) a Cross-modal Targets Generator to generate multi-modal and search space, and (B) a Temporal Sentence Tracker to track multi-modal targets' behavior and to predict query-related segment.
  arXiv  Detail & Related papers  (2023-02-21T16:42:52Z)
• End-to-end Tracking with a Multi-query Transformer [96.13468602635082]
  Multiple-object tracking (MOT) is a challenging task that requires simultaneous reasoning about location, appearance, and identity of the objects in the scene over time. Our aim in this paper is to move beyond tracking-by-detection approaches, to class-agnostic tracking that performs well also for unknown object classes.
  arXiv  Detail & Related papers  (2022-10-26T10:19:37Z)
• Tracking by Joint Local and Global Search: A Target-aware Attention based Approach [63.50045332644818]
  We propose a novel target-aware attention mechanism (termed TANet) to conduct joint local and global search for robust tracking. Specifically, we extract the features of target object patch and continuous video frames, then we track and feed them into a decoder network to generate target-aware global attention maps. In the tracking procedure, we integrate the target-aware attention with multiple trackers by exploring candidate search regions for robust tracking.
  arXiv  Detail & Related papers  (2021-06-09T06:54:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.