Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation

• URL: http://arxiv.org/abs/2604.01974v1
• Date: Thu, 02 Apr 2026 12:33:27 GMT
• Title: Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation
• Abstract summary: We introduce Interactive Tracking, a new paradigm that allows users to guide the tracker at any time using natural language commands.<n>We present InteractTrack, the first large-scale benchmark for interactive tracking, containing 150 videos with dense bounding box annotations and timestamped language instructions.<n>Third, we introduce Interactive Memory-Augmented Tracking (IMAT), a new baseline that employs a dynamic memory mechanism to learn from user feedback and update tracking behavior accordingly.
• Score: 60.96010213186819
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing visual trackers mainly operate in a non-interactive, fire-and-forget manner, making them impractical for real-world scenarios that require human-in-the-loop adaptation. To overcome this limitation, we introduce Interactive Tracking, a new paradigm that allows users to guide the tracker at any time using natural language commands. To support research in this direction, we make three main contributions. First, we present InteractTrack, the first large-scale benchmark for interactive tracking, containing 150 videos with dense bounding box annotations and timestamped language instructions. Second, we propose a comprehensive evaluation protocol and evaluate 25 representative trackers, showing that state-of-the-art methods fail in interactive scenarios; strong performance on conventional benchmarks does not transfer. Third, we introduce Interactive Memory-Augmented Tracking (IMAT), a new baseline that employs a dynamic memory mechanism to learn from user feedback and update tracking behavior accordingly. Our benchmark, protocol, and baseline establish a foundation for developing more intelligent, adaptive, and collaborative tracking systems, bridging the gap between automated perception and human guidance. The full benchmark, tracking results, and analysis are available at https://github.com/NorahGreen/InteractTrack.git.

Related papers

• QTrack: Query-Driven Reasoning for Multi-modal MOT [1.9981885081131854]
  Multi-object tracking (MOT) has traditionally focused on estimating of all objects in a video, without selectively reasoning about user-specified targets.<n>We introduce a query-driven tracking paradigm that formulates tracking as atemporal reasoning problem conditioned on natural language queries.<n>We present QTrack, an end-to-end vision-language model that integrates multimodal reasoning with tracking-oriented localization.
  arXiv  Detail & Related papers  (2026-03-14T05:11:36Z)
• GenTrack: A New Generation of Multi-Object Tracking [3.259045978275386]
  This paper introduces a novel multi-object tracking (MOT) method, dubbed GenTrack.<n>It employs both flexible and deterministic tracking manners to robustly handle unknown and time-varying numbers of targets.<n>GenTrack provides superior performance on standard benchmarks and real-world scenarios compared to state-of-the-art trackers.
  arXiv  Detail & Related papers  (2025-10-28T13:13:20Z)
• RSRNav: Reasoning Spatial Relationship for Image-Goal Navigation [57.197881161006904]
  Recent image-goal navigation (ImageNav) methods learn a perception-action policy by separately capturing semantic features of the goal and egocentric images.<n>We propose RSRNav, a simple yet effective method that reasons spatial relationships between the goal and current observations as navigation guidance.
  arXiv  Detail & Related papers  (2025-04-25T00:22:17Z)
• Divide and Merge: Motion and Semantic Learning in End-to-End Autonomous Driving [7.620469713146574]
  We propose Neural-Bayes motion decoding, a novel parallel detection, tracking, and prediction method.<n>We employ interactive semantic decoding to enhance information exchange in semantic tasks, promoting positive transfer.<n> Experiments on the nuScenes dataset with UniAD and SparseDrive confirm the effectiveness of our divide and merge approach.
  arXiv  Detail & Related papers  (2025-02-11T15:21:31Z)
• SynCL: A Synergistic Training Strategy with Instance-Aware Contrastive Learning for End-to-End Multi-Camera 3D Tracking [34.90147791481045]
  SynCL is a novel plug-and-play synergistic training strategy designed to co-facilitate multi-task learning for detection and tracking.<n>We propose a Task-specific Hybrid Matching module for a weight-shared cross-attention-based decoder.<n>We also introduce Instance-aware Contrastive Learning to break through the barrier of self-centric attention for track queries.
  arXiv  Detail & Related papers  (2024-11-11T08:18:49Z)
• I-MPN: Inductive Message Passing Network for Efficient Human-in-the-Loop Annotation of Mobile Eye Tracking Data [4.487146086221174]
  We present a novel human-centered learning algorithm designed for automated object recognition within mobile eye-tracking settings. Our approach seamlessly integrates an object detector with a spatial relation-aware inductive message-passing network (I-MPN), harnessing node profile information and capturing object correlations.
  arXiv  Detail & Related papers  (2024-06-10T13:08:31Z)
• Tracking with Human-Intent Reasoning [64.69229729784008]
  This work proposes a new tracking task -- Instruction Tracking. It involves providing implicit tracking instructions that require the trackers to perform tracking automatically in video frames. TrackGPT is capable of performing complex reasoning-based tracking.
  arXiv  Detail & Related papers  (2023-12-29T03:22:18Z)
• First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization [112.40598205054994]
  We formalize this idea as a completely unsupervised objective for optimizing interfaces. We conduct an observational study on 540K examples of users operating various keyboard and eye gaze interfaces for typing, controlling simulated robots, and playing video games. The results show that our mutual information scores are predictive of the ground-truth task completion metrics in a variety of domains.
  arXiv  Detail & Related papers  (2022-05-24T21:57:18Z)
• Learning Dynamic Compact Memory Embedding for Deformable Visual Object Tracking [82.34356879078955]
  We propose a compact memory embedding to enhance the discrimination of the segmentation-based deformable visual tracking method. Our method outperforms the excellent segmentation-based trackers, i.e., D3S and SiamMask on DAVIS 2017 benchmark.
  arXiv  Detail & Related papers  (2021-11-23T03:07:12Z)
• A Graph-based Interactive Reasoning for Human-Object Interaction Detection [71.50535113279551]
  We present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs. We construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Our framework is end-to-end trainable and free from costly annotations like human pose.
  arXiv  Detail & Related papers  (2020-07-14T09:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.