Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

• URL: http://arxiv.org/abs/2606.29357v1
• Date: Sun, 28 Jun 2026 12:12:18 GMT
• Title: Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking
• Abstract summary: We propose a novel language dependency parsing mechanism to distill tracking principal components.<n>We exploit the powerful cross-modal understanding capability of the pre-trained vision-language model Qwen-VL.<n>Our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks.
• Score: 61.91472604294714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language tracking guided by natural language specifications leverages high-level semantic cues of target objects to substantially boost tracking accuracy and robustness. Existing studies have verified that adaptively optimizing textual descriptions throughout the tracking process can effectively mitigate the semantic-visual mismatch induced by dynamic variations in target appearance, position, and other inherent attributes. Nevertheless, mainstream methods that directly generate textual information via sequence models or large language models inevitably suffer from inherent defects, including erroneous target updating, excessive background distraction, and pervasive hallucination artifacts. To address the aforementioned limitations, this paper proposes a novel language dependency parsing mechanism to precisely distill core tracking principal components, encompassing target objects, semantic concepts, and background contextual information. On this basis, we perform component-aware adaptive textual description updates by exploiting the powerful cross-modal understanding capability of the pre-trained vision-language model Qwen-VL. By integrating the proposed elaborately designed modules into the baseline framework, our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks, including TNL2K, LaSOT, TNLLT, and OTB-LANG. The source code and pre-trained models will be released at https://github.com/Event-AHU/Open_VLTrack.

Related papers

• Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking [21.780462979089886]
  LVTrack is a pure transformer framework that introduces a mode-conditioned Gated Feature to adaptively regulate textual guidance.<n>It directly harnesses a frozen vision-language pretrained model, greatly reducing training cost and preserving strong language understanding.<n>To further improve temporal localization, LVTrack integrates hybrid relative-absolute positional encodings with a lightweight memory mechanism.
  arXiv  Detail & Related papers  (2026-08-29T08:09:08Z)
• In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use [18.507118969955666]
  Vision-Language-Action models imitate expert action chunks conditioned on a static image and a fixed instruction.<n>We introduce textbfourmethod, a framework that endows a VLA with language competence through in-context post-training.<n>Our method consistently achieves SOTA results in both performance and efficiency when compared with CoT-based approaches.
  arXiv  Detail & Related papers  (2026-08-06T08:21:02Z)
• TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigation [70.23578202012048]
  Vision-Language Navigation (VLN) presents a unique challenge for Large Vision-Language Models (VLMs) due to their inherent architectural mismatch.<n>We propose TagaVLM (Topology-Aware Global Action reasoning), an end-to-end framework that explicitly injects topological structures into the VLM backbone.<n>To enhance topological node information, an Interleaved Navigation Prompt strengthens node-level visual-text alignment.<n>With the embedded topological graph, the model is capable of global action reasoning, allowing for robust path correction.
  arXiv  Detail & Related papers  (2026-03-03T13:28:07Z)
• ReasoningTrack: Chain-of-Thought Reasoning for Long-term Vision-Language Tracking [18.491855733401742]
  This paper proposes a novel reasoning-based vision-language tracking framework, named ReasoningTrack, based on a pre-trained vision-language model Qwen2.5-VL.<n>We embed the updated language descriptions and feed them into a unified tracking backbone network together with vision features.<n>In addition, we propose a large-scale long-term vision-language tracking benchmark dataset, termed TNLLT, which contains 200 video sequences.
  arXiv  Detail & Related papers  (2025-08-07T10:02:07Z)
• ATCTrack: Aligning Target-Context Cues with Dynamic Target States for Robust Vision-Language Tracking [0.6143225301480709]
  Vision-language tracking aims to locate the target object in the video sequence using a template patch and a language description provided in the initial frame.<n>To achieve robust tracking, it is essential not only to characterize the target features but also to utilize the context features related to the target.<n>We present a novel tracker named ATCTrack, which can obtain multimodal cues Aligned with the dynamic target states.
  arXiv  Detail & Related papers  (2025-07-26T09:05:12Z)
• CLDTracker: A Comprehensive Language Description for Visual Tracking [17.858934583542325]
  We propose CLDTracker, a novel Comprehensive Language Description framework for robust visual tracking.<n>Our tracker introduces a dual-branch architecture consisting of a textual and a visual branch.<n> Experiments on six standard VOT benchmarks demonstrate that CLDTracker achieves SOTA performance.
  arXiv  Detail & Related papers  (2025-05-29T17:39:30Z)
• Dynamic Updates for Language Adaptation in Visual-Language Tracking [10.64409248365897]
  We propose a vision-language tracking framework, named DUTrack, which captures the latest state of the target by dynamically updating multi-modal references to maintain consistency.<n>DUTrack achieves new state-of-the-art performance on four mainstream vision-language and two vision-only tracking benchmarks, including LaSOT, LaSOT$_rmext$, TNL2K, OTB99-Lang, GOT-10K, and UAV123.
  arXiv  Detail & Related papers  (2025-03-09T13:47:19Z)
• OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction [95.6266030753644]
  Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions.<n>Existing approaches require fine-tuning pre-trained vision-language models (VLMs) as visual and language features are independently fed into downstream policies.<n>We propose OTTER, a novel VLA architecture that leverages existing alignments through explicit, text-aware visual feature extraction.
  arXiv  Detail & Related papers  (2025-03-05T18:44:48Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• Multi-Granularity Language-Guided Training for Multi-Object Tracking [95.91263758294154]
  We propose a new multi-object tracking framework, named LG-MOT, that explicitly leverages language information at different levels of granularity.<n>At inference, our LG-MOT uses the standard visual features without relying on annotated language descriptions.<n>Our LG-MOT achieves an absolute gain of 2.2% in terms of target object association (IDF1 score) compared to the baseline using only visual features.
  arXiv  Detail & Related papers  (2024-06-07T11:18:40Z)
• Context-Aware Integration of Language and Visual References for Natural Language Tracking [27.3884348078998]
  Tracking by natural language specification (TNL) aims to consistently localize a target in a video sequence given a linguistic description in the initial frame. We propose a joint multi-modal tracking framework with 1) a prompt module to leverage the complement between temporal visual templates and language expressions, enabling precise and context-aware appearance and linguistic cues. This design ensures-temporal consistency by leveraging historical visual information and an integrated solution, generating predictions in a single step.
  arXiv  Detail & Related papers  (2024-03-29T04:58:33Z)
• Towards Unified Token Learning for Vision-Language Tracking [65.96561538356315]
  We present a vision-language (VL) tracking pipeline, termed textbfMMTrack, which casts VL tracking as a token generation task. Our proposed framework serializes language description and bounding box into a sequence of discrete tokens. In this new design paradigm, all token queries are required to perceive the desired target and directly predict spatial coordinates of the target.
  arXiv  Detail & Related papers  (2023-08-27T13:17:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.