USS: Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning

• URL: http://arxiv.org/abs/2606.25880v1
• Date: Wed, 24 Jun 2026 14:25:05 GMT
• Title: USS: Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning
• Abstract summary: Embodied Visual Tracking (EVT) requires an agent to continuously follow a specified target while actively moving through dynamic environments.<n>We propose a paradigm shift, reframing target indication in EVT from text-only specification to unified spatial-semantic prompting.<n>We introduce Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning.
• Score: 18.003030497082623
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied Visual Tracking (EVT) requires an agent to continuously follow a specified target while actively moving through dynamic environments. However, prevailing EVT paradigms predominantly rely on language-based target indication. While language is expressive and convenient, cluttered scenes often contain multiple objects that satisfy the same semantic description, leading to ambiguous target grounding. We therefore propose a paradigm shift, reframing target indication in EVT from text-only specification to unified spatial-semantic prompting. Based on this paradigm, we introduce Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning, USS, an end-to-end embodied tracking framework that supports text, point, bounding box, and mask prompts within a unified architecture. USS encodes heterogeneous prompts with modality-specific encoders, fuses prompt tokens with visual features through hybrid attention, and decodes compact prompt-conditioned representations into egocentric waypoints. To further improve temporal robustness, USS incorporates a latent world model that predicts future representations through self-supervised alignment. Real-robot experiments demonstrate that explicit spatial target cues yield higher success rates than text-only prompts, particularly in scenarios involving similar distractors and longer-horizon tracking where maintaining instance-level target identity is critical. In the simulation benchmark, USS also achieves state-of-the-art performance among non-MLLM-based methods and competitive results against recent MLLM-based approaches with faster inference speed. Our findings reveal that spatial-semantic prompting provides a more precise and flexible target indication interface for embodied visual tracking. Project site: https://arescheah.github.io/uss-project-page/.

Related papers

• One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spotting [51.09550363815034]
  Scene text spotting requires high-precision alignment between textual recognition and spatial localization.<n>We propose Single-Patch Text Spotting (SPaTS), a vision-centric framework that routes each text instance through a single anchor visual token.<n>We show that SPaTS consistently and significantly outperforms both frontier closed-source MLLMs and OCR MLLMs.
  arXiv  Detail & Related papers  (2026-07-30T09:17:48Z)
• ReferTrack: Referring Then Tracking for Embodied Visual Tracking [13.368122578638847]
  Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language.<n>We introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera.<n>Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision.
  arXiv  Detail & Related papers  (2026-07-22T12:05:13Z)
• Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking [61.91472604294714]
  We propose a novel language dependency parsing mechanism to distill tracking principal components.<n>We exploit the powerful cross-modal understanding capability of the pre-trained vision-language model Qwen-VL.<n>Our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks.
  arXiv  Detail & Related papers  (2026-06-28T12:12:18Z)
• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
  We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
  arXiv  Detail & Related papers  (2026-05-26T00:08:06Z)
• See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding [41.70825455674463]
  SWIM (See What I Mean) is a training strategy that aligns vision and language representations to enable fine-grained object understanding solely from textual prompts.<n>We show that SWIM substantially improves text-visual alignment and achieves superior performance over visual-prompt-based methods on fine-grained object understanding benchmarks.
  arXiv  Detail & Related papers  (2026-05-18T08:09:37Z)
• Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining [59.2578488860426]
  Heterogeneous multi-modal remote sensing object detection aims to accurately detect objects from diverse sensors.<n>Existing approaches largely adopt a late alignment paradigm, in which modality alignment and task-specific optimization are entangled during downstream fine-tuning.<n>We propose BabelRS, a unified language-pivoted pretraining framework that explicitly decouples modality alignment from downstream task learning.
  arXiv  Detail & Related papers  (2026-03-02T11:38:12Z)
• SGHA-Attack: Semantic-Guided Hierarchical Alignment for Transferable Targeted Attacks on Vision-Language Models [73.19044613922911]
  Large vision-language models (VLMs) are vulnerable to transfer-based adversarial perturbations.<n>We propose SGHA-Attack, a framework that adopts multiple target references and enforces intermediate-layer consistency.<n>Experiments on open-source and commercial black-box VLMs show that SGHA-Attack achieves stronger targeted transferability than prior methods.
  arXiv  Detail & Related papers  (2026-02-02T03:10:41Z)
• VPTracker: Global Vision-Language Tracking via Visual Prompt and MLLM [45.56517073754981]
  Vision-Language Tracking aims to continuously localize objects described by a visual template and a language description.<n>Existing methods, however, are typically limited to local search, making them prone to failures under viewpoint changes.<n>We introduce the first global tracking framework based on Multimodal Large Language Models (VPTracker), exploiting their powerful semantic reasoning to locate targets.
  arXiv  Detail & Related papers  (2025-12-28T06:12:28Z)
• EPIPTrack: Rethinking Prompt Modeling with Explicit and Implicit Prompts for Multi-Object Tracking [10.065921746316642]
  We propose a unified vision-language tracking framework, named EPIPTrack.<n>EPIPTrack leverages explicit and implicit prompts for dynamic target modeling and semantic alignment.<n>Experiments on MOT17, MOT20, and Dance demonstrate that EPIPTrack outperforms existing trackers in diverse scenarios.
  arXiv  Detail & Related papers  (2025-10-15T07:39:30Z)
• A Multimodal Depth-Aware Method For Embodied Reference Understanding [56.30142869506262]
  Embodied Reference Understanding requires identifying a target object in a visual scene based on both language instructions and pointing cues.<n>We propose a novel ERU framework that jointly leverages data augmentation, depth-map modality, and a depth-aware decision module.
  arXiv  Detail & Related papers  (2025-10-09T14:32:21Z)
• ATCTrack: Aligning Target-Context Cues with Dynamic Target States for Robust Vision-Language Tracking [0.6143225301480709]
  Vision-language tracking aims to locate the target object in the video sequence using a template patch and a language description provided in the initial frame.<n>To achieve robust tracking, it is essential not only to characterize the target features but also to utilize the context features related to the target.<n>We present a novel tracker named ATCTrack, which can obtain multimodal cues Aligned with the dynamic target states.
  arXiv  Detail & Related papers  (2025-07-26T09:05:12Z)
• Hierarchical Instruction-aware Embodied Visual Tracking [35.73851196966425]
  User-Centric Embodied Visual Tracking (UC-EVT) presents a novel challenge for reinforcement learning-based models.<n>We propose textbf Instruction-aware Embodied Visual Tracking (HIEVT) agent, which bridges instruction comprehension and action generation using textitspatial goals as intermediaries.
  arXiv  Detail & Related papers  (2025-05-27T04:36:26Z)
• Towards Unified Token Learning for Vision-Language Tracking [65.96561538356315]
  We present a vision-language (VL) tracking pipeline, termed textbfMMTrack, which casts VL tracking as a token generation task. Our proposed framework serializes language description and bounding box into a sequence of discrete tokens. In this new design paradigm, all token queries are required to perceive the desired target and directly predict spatial coordinates of the target.
  arXiv  Detail & Related papers  (2023-08-27T13:17:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.