ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model

• URL: http://arxiv.org/abs/2411.01756v1
• Date: Mon, 04 Nov 2024 02:43:55 GMT
• Title: ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model
• Authors: Yiming Sun, Fan Yu, Shaoxiang Chen, Yu Zhang, Junwei Huang, Chenhui Li, Yang Li, Changbo Wang,
• Abstract summary: Vision-Language(VL) trackers have proposed to utilize additional natural language descriptions to enhance versatility in various applications. VL trackers are still inferior to State-of-The-Art (SoTA) visual trackers in terms of tracking performance. We propose ChatTracker to leverage the wealth of world knowledge in the Multimodal Large Language Model (MLLM) to generate high-quality language descriptions.
• Score: 29.702895846058265
• License:
• Abstract: Visual object tracking aims to locate a targeted object in a video sequence based on an initial bounding box. Recently, Vision-Language~(VL) trackers have proposed to utilize additional natural language descriptions to enhance versatility in various applications. However, VL trackers are still inferior to State-of-The-Art (SoTA) visual trackers in terms of tracking performance. We found that this inferiority primarily results from their heavy reliance on manual textual annotations, which include the frequent provision of ambiguous language descriptions. In this paper, we propose ChatTracker to leverage the wealth of world knowledge in the Multimodal Large Language Model (MLLM) to generate high-quality language descriptions and enhance tracking performance. To this end, we propose a novel reflection-based prompt optimization module to iteratively refine the ambiguous and inaccurate descriptions of the target with tracking feedback. To further utilize semantic information produced by MLLM, a simple yet effective VL tracking framework is proposed and can be easily integrated as a plug-and-play module to boost the performance of both VL and visual trackers. Experimental results show that our proposed ChatTracker achieves a performance comparable to existing methods.

Related papers

• VOVTrack: Exploring the Potentiality in Videos for Open-Vocabulary Object Tracking [61.56592503861093]
  This issue amalgamates the complexities of open-vocabulary object detection (OVD) and multi-object tracking (MOT) Existing approaches to OVMOT often merge OVD and MOT methodologies as separate modules, predominantly focusing on the problem through an image-centric lens. We propose VOVTrack, a novel method that integrates object states relevant to MOT and video-centric training to address this challenge from a video object tracking standpoint.
  arXiv  Detail & Related papers  (2024-10-11T05:01:49Z)
• DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM [23.551036494221222]
  We propose a new visual language tracking benchmark with diverse texts, named DTVLT, based on five prominent VLT and SOT benchmarks. We offer four texts in our benchmark, considering the extent and density of semantic information. We conduct comprehensive experimental analyses on DTVLT, evaluating the impact of diverse text on tracking performance.
  arXiv  Detail & Related papers  (2024-10-03T13:57:07Z)
• Multi-Granularity Language-Guided Multi-Object Tracking [95.91263758294154]
  We propose a new multi-object tracking framework, named LG-MOT, that explicitly leverages language information at different levels of granularity. At inference, our LG-MOT uses the standard visual features without relying on annotated language descriptions. Our LG-MOT achieves an absolute gain of 2.2% in terms of target object association (IDF1 score) compared to the baseline using only visual features.
  arXiv  Detail & Related papers  (2024-06-07T11:18:40Z)
• DTLLM-VLT: Diverse Text Generation for Visual Language Tracking Based on LLM [23.551036494221222]
  Visual Language Tracking (VLT) enhances single object tracking (SOT) by integrating natural language descriptions from a video, for the precise tracking of a specified object. Most VLT benchmarks are annotated in a single granularity and lack a coherent semantic framework to provide scientific guidance. We introduce DTLLM-VLT, which automatically generates extensive and multi-granularity text to enhance environmental diversity.
  arXiv  Detail & Related papers  (2024-05-20T16:01:01Z)
• Unifying Visual and Vision-Language Tracking via Contrastive Learning [34.49865598433915]
  Single object tracking aims to locate the target object in a video sequence according to different modal references. Due to the gap between different modalities, most existing trackers are designed for single or partial of these reference settings. We present a unified tracker called UVLTrack, which can simultaneously handle all three reference settings.
  arXiv  Detail & Related papers  (2024-01-20T13:20:54Z)
• Tracking with Human-Intent Reasoning [64.69229729784008]
  This work proposes a new tracking task -- Instruction Tracking. It involves providing implicit tracking instructions that require the trackers to perform tracking automatically in video frames. TrackGPT is capable of performing complex reasoning-based tracking.
  arXiv  Detail & Related papers  (2023-12-29T03:22:18Z)
• Beyond Visual Cues: Synchronously Exploring Target-Centric Semantics for Vision-Language Tracking [3.416427651955299]
  Single object tracking aims to locate one specific target in video sequences, given its initial state. Vision-Language (VL) tracking has emerged as a promising approach. We present a novel tracker that progressively explores target-centric semantics for VL tracking.
  arXiv  Detail & Related papers  (2023-11-28T02:28:12Z)
• Towards Unified Token Learning for Vision-Language Tracking [65.96561538356315]
  We present a vision-language (VL) tracking pipeline, termed textbfMMTrack, which casts VL tracking as a token generation task. Our proposed framework serializes language description and bounding box into a sequence of discrete tokens. In this new design paradigm, all token queries are required to perceive the desired target and directly predict spatial coordinates of the target.
  arXiv  Detail & Related papers  (2023-08-27T13:17:34Z)
• Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models [50.07056960586183]
  We propose Position-enhanced Visual Instruction Tuning (PVIT) to extend the functionality of Multimodal Large Language Models (MLLMs) This integration promotes a more detailed comprehension of images for the MLLM. We present both quantitative experiments and qualitative analysis that demonstrate the superiority of the proposed model.
  arXiv  Detail & Related papers  (2023-08-25T15:33:47Z)
• Divert More Attention to Vision-Language Object Tracking [87.31882921111048]
  We argue that the lack of large-scale vision-language annotated videos and ineffective vision-language interaction learning motivate us to design more effective vision-language representation for tracking. Particularly, in this paper, we first propose a general attribute annotation strategy to decorate videos in six popular tracking benchmarks, which contributes a large-scale vision-language tracking database with more than 23,000 videos. We then introduce a novel framework to improve tracking by learning a unified-adaptive VL representation, where the cores are the proposed asymmetric architecture search and modality mixer (ModaMixer)
  arXiv  Detail & Related papers  (2023-07-19T15:22:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.