ReferTrack: Referring Then Tracking for Embodied Visual Tracking

• URL: http://arxiv.org/abs/2607.20061v1
• Date: Wed, 22 Jul 2026 12:05:13 GMT
• Title: ReferTrack: Referring Then Tracking for Embodied Visual Tracking
• Abstract summary: Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language.<n>We introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera.<n>Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision.
• Score: 13.368122578638847
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.

Related papers

• USS: Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning [18.003030497082623]
  Embodied Visual Tracking (EVT) requires an agent to continuously follow a specified target while actively moving through dynamic environments.<n>We propose a paradigm shift, reframing target indication in EVT from text-only specification to unified spatial-semantic prompting.<n>We introduce Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning.
  arXiv  Detail & Related papers  (2026-06-24T14:25:05Z)
• QTrack: Query-Driven Reasoning for Multi-modal MOT [1.9981885081131854]
  Multi-object tracking (MOT) has traditionally focused on estimating of all objects in a video, without selectively reasoning about user-specified targets.<n>We introduce a query-driven tracking paradigm that formulates tracking as atemporal reasoning problem conditioned on natural language queries.<n>We present QTrack, an end-to-end vision-language model that integrates multimodal reasoning with tracking-oriented localization.
  arXiv  Detail & Related papers  (2026-03-14T05:11:36Z)
• Walker: Self-supervised Multiple Object Tracking by Walking on Temporal Appearance Graphs [117.67620297750685]
  We introduce Walker, the first self-supervised tracker that learns from videos with sparse bounding box annotations, and no tracking labels. Walker is the first self-supervised tracker to achieve competitive performance on MOT17, DanceTrack, and BDD100K.
  arXiv  Detail & Related papers  (2024-09-25T18:00:00Z)
• STCMOT: Spatio-Temporal Cohesion Learning for UAV-Based Multiple Object Tracking [13.269416985959404]
  Multiple object tracking (MOT) in Unmanned Aerial Vehicle (UAV) videos is important for diverse applications in computer vision. We propose a novel Spatio-Temporal Cohesion Multiple Object Tracking framework (STCMOT) We use historical embedding features to model the representation of ReID and detection features in a sequential order. Our framework sets a new state-of-the-art performance in MOTA and IDF1 metrics.
  arXiv  Detail & Related papers  (2024-09-17T14:34:18Z)
• BEVTrack: A Simple and Strong Baseline for 3D Single Object Tracking in Bird's-Eye View [54.48052449493636]
  3D Single Object Tracking (SOT) is a fundamental task in computer vision and plays a critical role in applications like autonomous driving.<n>We propose BEVTrack, a simple yet effective motion-based tracking method.<n>We show that BEVTrack achieves state-of-the-art results while operating at 200 FPS, enabling real-time applicability.
  arXiv  Detail & Related papers  (2023-09-05T12:42:26Z)
• OmniTracker: Unifying Object Tracking by Tracking-with-Detection [119.51012668709502]
  OmniTracker is presented to resolve all the tracking tasks with a fully shared network architecture, model weights, and inference pipeline. Experiments on 7 tracking datasets, including LaSOT, TrackingNet, DAVIS16-17, MOT17, MOTS20, and YTVIS19, demonstrate that OmniTracker achieves on-par or even better results than both task-specific and unified tracking models.
  arXiv  Detail & Related papers  (2023-03-21T17:59:57Z)
• DIVOTrack: A Novel Dataset and Baseline Method for Cross-View Multi-Object Tracking in DIVerse Open Scenes [74.64897845999677]
  We introduce a new cross-view multi-object tracking dataset for DIVerse Open scenes with dense tracking pedestrians. Our DIVOTrack has fifteen distinct scenarios and 953 cross-view tracks, surpassing all cross-view multi-object tracking datasets currently available. Furthermore, we provide a novel baseline cross-view tracking method with a unified joint detection and cross-view tracking framework named CrossMOT.
  arXiv  Detail & Related papers  (2023-02-15T14:10:42Z)
• Learning Dynamic Compact Memory Embedding for Deformable Visual Object Tracking [82.34356879078955]
  We propose a compact memory embedding to enhance the discrimination of the segmentation-based deformable visual tracking method. Our method outperforms the excellent segmentation-based trackers, i.e., D3S and SiamMask on DAVIS 2017 benchmark.
  arXiv  Detail & Related papers  (2021-11-23T03:07:12Z)
• Learning to Track with Object Permanence [61.36492084090744]
  We introduce an end-to-end trainable approach for joint object detection and tracking. Our model, trained jointly on synthetic and real data, outperforms the state of the art on KITTI, and MOT17 datasets.
  arXiv  Detail & Related papers  (2021-03-26T04:43:04Z)
• TDIOT: Target-driven Inference for Deep Video Object Tracking [0.2457872341625575]
  In this work, we adopt the pre-trained Mask R-CNN deep object detector as the baseline. We introduce a novel inference architecture placed on top of FPN-ResNet101 backbone of Mask R-CNN to jointly perform detection and tracking. The proposed single object tracker, TDIOT, applies an appearance similarity-based temporal matching for data association.
  arXiv  Detail & Related papers  (2021-03-19T20:45:06Z)
• Learning Spatio-Appearance Memory Network for High-Performance Visual Tracking [79.80401607146987]
  Existing object tracking usually learns a bounding-box based template to match visual targets across frames, which cannot accurately learn a pixel-wise representation. This paper presents a novel segmentation-based tracking architecture, which is equipped with a local-temporal memory network to learn accurate-temporal correspondence.
  arXiv  Detail & Related papers  (2020-09-21T08:12:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.