Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers

• URL: http://arxiv.org/abs/2407.08394v2
• Date: Tue, 16 Jul 2024 07:34:53 GMT
• Title: Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers
• Abstract summary: Diff-Tracker is a novel approach for the challenging unsupervised visual tracking task. Our main idea is to leverage the rich knowledge encapsulated within the pre-trained diffusion model.
• Score: 19.446590457828012
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Diff-Tracker, a novel approach for the challenging unsupervised visual tracking task leveraging the pre-trained text-to-image diffusion model. Our main idea is to leverage the rich knowledge encapsulated within the pre-trained diffusion model, such as the understanding of image semantics and structural information, to address unsupervised visual tracking. To this end, we design an initial prompt learner to enable the diffusion model to recognize the tracking target by learning a prompt representing the target. Furthermore, to facilitate dynamic adaptation of the prompt to the target's movements, we propose an online prompt updater. Extensive experiments on five benchmark datasets demonstrate the effectiveness of our proposed method, which also achieves state-of-the-art performance.

Related papers

• Leveraging Text-to-Image Diffusion Models for Unsupervised Visual Object Tracking [53.08526613328243]
  Unsupervised visual object tracking is a challenging task that requires following arbitrary targets in videos without training on ground-truth annotations.<n>We build on text-to-image diffusion models to exploit the rich semantic knowledge encoded in pretrained models.<n>Our method Diff-Tracking is composed of two main components: an initial prompt learner and an online prompt updater.
  arXiv  Detail & Related papers  (2026-05-26T12:26:56Z)
• Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision [80.2787127430925]
  Distinguishing visually similar objects by their motion remains a critical challenge in computer vision.<n>We find that pre-trained video diffusion models inherently learn motion representations suitable for tracking without task-specific training.<n>Our method achieves up to a 6-point improvement over recent self-supervised approaches on established benchmarks.
  arXiv  Detail & Related papers  (2025-12-02T02:17:34Z)
• Preserving Clusters in Prompt Learning for Unsupervised Domain Adaptation [29.809079908218607]
  This work introduces a fresh solution to reinforce base pseudo-labels and facilitate target-prompt learning.<n>We first propose to leverage the reference predictions based on the relationship between source and target visual embeddings.<n>We later show that there is a strong clustering behavior observed between visual and text embeddings in pre-trained multi-modal models.
  arXiv  Detail & Related papers  (2025-06-13T06:33:27Z)
• InstaRevive: One-Step Image Enhancement via Dynamic Score Matching [66.97989469865828]
  InstaRevive is an image enhancement framework that employs score-based diffusion distillation to harness potent generative capability. Our framework delivers high-quality and visually appealing results across a diverse array of challenging tasks and datasets.
  arXiv  Detail & Related papers  (2025-04-22T01:19:53Z)
• DINTR: Tracking via Diffusion-based Interpolation [12.130669304428565]
  This work proposes a novel diffusion-based methodology to formulate the tracking task. Our INterpolation TrackeR (DINTR) presents a promising new paradigm and achieves a superior multiplicity on seven benchmarks across five indicator representations.
  arXiv  Detail & Related papers  (2024-10-14T00:41:58Z)
• Harnessing Diffusion Models for Visual Perception with Meta Prompts [68.78938846041767]
  We propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. We introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes.
  arXiv  Detail & Related papers  (2023-12-22T14:40:55Z)
• CiteTracker: Correlating Image and Text for Visual Tracking [114.48653709286629]
  We propose the CiteTracker to enhance target modeling and inference in visual tracking by connecting images and text. Specifically, we develop a text generation module to convert the target image patch into a descriptive text. We then associate the target description and the search image using an attention-based correlation module to generate the correlated features for target state reference.
  arXiv  Detail & Related papers  (2023-08-22T09:53:12Z)
• Free-ATM: Exploring Unsupervised Learning on Diffusion-Generated Images with Free Attention Masks [64.67735676127208]
  Text-to-image diffusion models have shown great potential for benefiting image recognition. Although promising, there has been inadequate exploration dedicated to unsupervised learning on diffusion-generated images. We introduce customized solutions by fully exploiting the aforementioned free attention masks.
  arXiv  Detail & Related papers  (2023-08-13T10:07:46Z)
• Unleashing Text-to-Image Diffusion Models for Visual Perception [84.41514649568094]
  VPD (Visual Perception with a pre-trained diffusion model) is a new framework that exploits the semantic information of a pre-trained text-to-image diffusion model in visual perception tasks. We show that VPD can be faster adapted to downstream visual perception tasks using the proposed VPD.
  arXiv  Detail & Related papers  (2023-03-03T18:59:47Z)
• Denoising Diffusion Probabilistic Models for Generation of Realistic Fully-Annotated Microscopy Image Data Sets [1.07539359851877]
  In this study, we demonstrate that diffusion models can effectively generate fully-annotated microscopy image data sets. The proposed pipeline helps to reduce the reliance on manual annotations when training deep learning-based segmentation approaches.
  arXiv  Detail & Related papers  (2023-01-02T14:17:08Z)
• Towards Sequence-Level Training for Visual Tracking [60.95799261482857]
  This work introduces a sequence-level training strategy for visual tracking based on reinforcement learning. Four representative tracking models, SiamRPN++, SiamAttn, TransT, and TrDiMP, consistently improve by incorporating the proposed methods in training.
  arXiv  Detail & Related papers  (2022-08-11T13:15:36Z)
• Learning Actor-centered Representations for Action Localization in Streaming Videos using Predictive Learning [18.757368441841123]
  Event perception tasks such as recognizing and localizing actions in streaming videos are essential for tackling visual understanding tasks. We tackle the problem of learning textitactor-centered representations through the notion of continual hierarchical predictive learning. Inspired by cognitive theories of event perception, we propose a novel, self-supervised framework.
  arXiv  Detail & Related papers  (2021-04-29T06:06:58Z)
• Self-supervised Video Object Segmentation [76.83567326586162]
  The objective of this paper is self-supervised representation learning, with the goal of solving semi-supervised video object segmentation (a.k.a. dense tracking) We make the following contributions: (i) we propose to improve the existing self-supervised approach, with a simple, yet more effective memory mechanism for long-term correspondence matching; (ii) by augmenting the self-supervised approach with an online adaptation module, our method successfully alleviates tracker drifts caused by spatial-temporal discontinuity; (iv) we demonstrate state-of-the-art results among the self-supervised approaches on DAVIS-2017 and YouTube
  arXiv  Detail & Related papers  (2020-06-22T17:55:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.