Leveraging Text-to-Image Diffusion Models for Unsupervised Visual Object Tracking

• URL: http://arxiv.org/abs/2605.26933v1
• Date: Tue, 26 May 2026 12:26:56 GMT
• Title: Leveraging Text-to-Image Diffusion Models for Unsupervised Visual Object Tracking
• Abstract summary: Unsupervised visual object tracking is a challenging task that requires following arbitrary targets in videos without training on ground-truth annotations.<n>We build on text-to-image diffusion models to exploit the rich semantic knowledge encoded in pretrained models.<n>Our method Diff-Tracking is composed of two main components: an initial prompt learner and an online prompt updater.
• Score: 53.08526613328243
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Unsupervised visual object tracking is a challenging task that requires following arbitrary targets in videos without training on ground-truth annotations. Despite considerable progress, existing state-of-the-art unsupervised trackers often struggle in scenarios that demand fine-grained understanding of semantic and visual structural information within video frames. Text-to-image diffusion models are well known for their ability to generate images that accurately reflect the semantics and structures described in the input prompt, demonstrating a strong grasp of visual semantics and structures. Building on this capability, we approach the unsupervised tracking from a new perspective by exploiting the rich semantic knowledge encoded in pretrained text-to-image diffusion models. To adapt the diffusion models, which are originally developed for image generation, to the tracking task, we reinterpret the models as a bridge between text and image modalities. This connection is realized through the cross-attention mechanism: when both text and an image are input into the models, they highlight the regions of the image that are semantically aligned with the text in the cross-attention maps. We therefore learn a prompt that represents the tracking target and activates its corresponding region in the cross-attention map for each frame, which enables object tracking with the diffusion model. Specifically, our method Diff-Tracking is composed of two main components: an initial prompt learner and an online prompt updater. The initial prompt learner generates a prompt that captures the target object in the first frame, allowing the diffusion model to identify the target. The online prompt updater refines the prompt based on motion information, enabling consistent tracking across video frames. We evaluate our approach on six challenging tracking datasets demonstrate the effectiveness of our approach.

Related papers

• Semantic-Structural Alignment for Generative Pictorial Charts [59.45629259524998]
  We present a generative framework for the automated synthesis of pictorial charts.<n>We frame the problem as a dual-conditioned generation task guided by two parallel external control signals.<n>Our method produces pictorial charts that are both artistically compelling and structurally consistent.
  arXiv  Detail & Related papers  (2026-05-05T05:20:46Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation [41.79836820271156]
  "In-Context Brush" is a zero-shot framework for customized subject insertion.<n>We formulate the object image and the textual prompts as cross-modal demonstrations.<n>The goal is to inpaint the target image with the subject aligning textual prompts without model tuning.
  arXiv  Detail & Related papers  (2025-05-26T17:49:10Z)
• DINTR: Tracking via Diffusion-based Interpolation [12.130669304428565]
  This work proposes a novel diffusion-based methodology to formulate the tracking task. Our INterpolation TrackeR (DINTR) presents a promising new paradigm and achieves a superior multiplicity on seven benchmarks across five indicator representations.
  arXiv  Detail & Related papers  (2024-10-14T00:41:58Z)
• Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers [19.446590457828012]
  Diff-Tracker is a novel approach for the challenging unsupervised visual tracking task. Our main idea is to leverage the rich knowledge encapsulated within the pre-trained diffusion model.
  arXiv  Detail & Related papers  (2024-07-11T10:57:33Z)
• Leveraging Open-Vocabulary Diffusion to Camouflaged Instance Segmentation [59.78520153338878]
  Text-to-image diffusion techniques have shown exceptional capability of producing high-quality images from text descriptions. We propose a method built upon a state-of-the-art diffusion model, empowered by open-vocabulary to learn multi-scale textual-visual features for camouflaged object representations.
  arXiv  Detail & Related papers  (2023-12-29T07:59:07Z)
• CiteTracker: Correlating Image and Text for Visual Tracking [114.48653709286629]
  We propose the CiteTracker to enhance target modeling and inference in visual tracking by connecting images and text. Specifically, we develop a text generation module to convert the target image patch into a descriptive text. We then associate the target description and the search image using an attention-based correlation module to generate the correlated features for target state reference.
  arXiv  Detail & Related papers  (2023-08-22T09:53:12Z)
• Helping Hands: An Object-Aware Ego-Centric Video Recognition Model [60.350851196619296]
  We introduce an object-aware decoder for improving the performance of ego-centric representations on ego-centric videos. We show that the model can act as a drop-in replacement for an ego-awareness video model to improve performance through visual-text grounding.
  arXiv  Detail & Related papers  (2023-08-15T17:58:11Z)
• Unleashing Text-to-Image Diffusion Models for Visual Perception [84.41514649568094]
  VPD (Visual Perception with a pre-trained diffusion model) is a new framework that exploits the semantic information of a pre-trained text-to-image diffusion model in visual perception tasks. We show that VPD can be faster adapted to downstream visual perception tasks using the proposed VPD.
  arXiv  Detail & Related papers  (2023-03-03T18:59:47Z)
• Directed Diffusion: Direct Control of Object Placement through Attention Guidance [15.275386705641266]
  Text-guided diffusion models can generate an effectively endless variety of images given only a short text prompt describing the desired image content. These models often struggle to compose scenes containing several key objects such as characters in specified positional relationships. In this work, we take a particularly straightforward approach to providing the needed direction.
  arXiv  Detail & Related papers  (2023-02-25T20:48:15Z)
• Semantic-Aware Fine-Grained Correspondence [8.29030327276322]
  We propose to learn semantic-aware fine-grained correspondence using image-level self-supervised methods. We design a pixel-level self-supervised learning objective which specifically targets fine-grained correspondence. Our method surpasses previous state-of-the-art self-supervised methods using convolutional networks on a variety of visual correspondence tasks.
  arXiv  Detail & Related papers  (2022-07-21T12:51:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.