Just Functioning as a Hook for Two-Stage Referring Multi-Object Tracking

• URL: http://arxiv.org/abs/2503.07516v3
• Date: Thu, 14 Aug 2025 05:26:18 GMT
• Title: Just Functioning as a Hook for Two-Stage Referring Multi-Object Tracking
• Authors: Weize Li, Yunhao Du, Qixiang Yin, Zhicheng Zhao, Fei Su, Daqi Liu,
• Abstract summary: Referring Multi-Object Tracking aims to localize target trajectories in videos specified by natural language expressions.<n>We present a systematic analysis of the intrinsic relationship between the two subtasks of tracking and referring in RMOT.<n>We propose JustHook, a novel two-stage RBT framework where a Hook module is firstly designed to redefine the linkage between subtasks.
• Score: 22.669740476582835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring Multi-Object Tracking (RMOT) aims to localize target trajectories in videos specified by natural language expressions. Despite recent progress, the intrinsic relationship between the two subtasks of tracking and referring in RMOT has not been fully studied. In this paper, we present a systematic analysis of their interdependence, revealing that current two-stage Referring-by-Tracking (RBT) frameworks remain fundamentally limited by insufficient modeling of subtask interactions and inflexible reliance on semantic alignment modules like CLIP. To this end, we propose JustHook, a novel two-stage RBT framework where a Hook module is firstly designed to redefine the linkage between subtasks. The Hook is built centered on grid sampling at the feature-level and is used for context-aware target feature extraction. Moreover, we propose a Parallel Combined Decoder (PCD) that learns in a unified joint feature space rather than relying on pre-defined cross-modal embeddings. Our design not only enhances the interpretability and modularity but also significantly improves the generalization. Extensive experiments on Refer-KITTI, Refer-KITTI-V2, and Refer-Dance demonstrate that JustHook achieves state-of-the-art performance, improving the HOTA by +6.9\% on Refer-KITTI-V2 with superior efficiency. Code will be available soon.

Related papers

• Toward Stable Semi-Supervised Remote Sensing Segmentation via Co-Guidance and Co-Fusion [31.189038928192648]
  Co2S is a semi-supervised RS segmentation framework that fuses priors from vision-language models and self-supervised models.<n>An explicit-implicit semantic co-guidance mechanism is introduced that utilizes text embeddings and learnable queries.<n>Experiments on six popular datasets demonstrate the superiority of the proposed method.
  arXiv  Detail & Related papers  (2025-12-28T18:24:19Z)
• Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition [36.662223760818584]
  Trokens is a novel approach that transforms trajectory points into semantic-aware relational tokens for action recognition.<n>We develop a motion modeling framework that captures both intra-trajectory dynamics through the Histogram of Oriented Displacements (HoD) and inter-trajectory relationships to model complex action patterns.<n>Our approach effectively combines these trajectory tokens with semantic features to enhance appearance features with motion information, achieving state-of-the-art performance across six diverse few-shot action recognition benchmarks.
  arXiv  Detail & Related papers  (2025-08-05T17:59:58Z)
• CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding [56.30142869506262]
  Embodied Reference Understanding involves predicting the object that a person in the scene is referring to through both pointing gesture and language.<n>We propose a dual-model framework, where one model learns from the head-to-fingertip direction and the other from the wrist-to-fingertip direction.<n>We validate our approach through extensive experiments and analysis on the benchmark YouRefIt dataset, achieving an improvement of approximately 4 mAP at the 0.25 IoU threshold.
  arXiv  Detail & Related papers  (2025-07-29T15:00:21Z)
• Enhanced Partially Relevant Video Retrieval through Inter- and Intra-Sample Analysis with Coherence Prediction [18.24629930062925]
  Partially Relevant Video Retrieval aims to retrieve the target video that is partially relevant to a text query.<n>Existing methods coarsely align paired videos and text queries to construct the semantic space.<n>We propose a novel PRVR framework to systematically exploit inter-sample correlation and intra-sample redundancy.
  arXiv  Detail & Related papers  (2025-04-28T09:52:46Z)
• DINOv2-powered Few-Shot Semantic Segmentation: A Unified Framework via Cross-Model Distillation and 4D Correlation Mining [30.564216896513596]
  Few-shot semantic segmentation has gained increasing interest due to its generalization capability.<n>Recent approaches have turned to foundation models to enhance representation transferability.<n>We propose FS-DINO, with only DINOv2's encoder and a lightweight segmenter.
  arXiv  Detail & Related papers  (2025-04-22T07:47:06Z)
• Cognitive Disentanglement for Referring Multi-Object Tracking [28.325814292139686]
  We propose a Cognitive Disentanglement for Referring Multi-Object Tracking (CDRMT) framework. CDRMT adapts the "what" and "where" pathways from the human visual processing system to RMOT tasks. Experiments on different benchmark datasets demonstrate that CDRMT achieves substantial improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-03-14T15:21:54Z)
• Col-OLHTR: A Novel Framework for Multimodal Online Handwritten Text Recognition [82.88856416080331]
  Online Handwritten Text Recognition (OLHTR) has gained considerable attention for its diverse range of applications.<n>Current approaches usually treat OLHTR as a sequence recognition task, employing either a single trajectory or image encoder, or multi-stream encoders.<n>We propose a Collaborative learning-based OLHTR framework, called Col-OLHTR, that learns multimodal features during training while maintaining a single-stream inference process.
  arXiv  Detail & Related papers  (2025-02-10T02:12:24Z)
• Cross-Modal Bidirectional Interaction Model for Referring Remote Sensing Image Segmentation [50.433911327489554]
  The goal of referring remote sensing image segmentation (RRSIS) is to generate a pixel-level mask of the target object identified by the referring expression.<n>To address the aforementioned challenges, a novel RRSIS framework is proposed, termed the cross-modal bidirectional interaction model (CroBIM)<n>To further forster the research of RRSIS, we also construct RISBench, a new large-scale benchmark dataset comprising 52,472 image-language-label triplets.
  arXiv  Detail & Related papers  (2024-10-11T08:28:04Z)
• Zero-Shot Skeleton-based Action Recognition with Dual Visual-Text Alignment [11.72557768532557]
  Key to zero-shot action recognition lies in aligning visual features with semantic vectors representing action categories. Our approach achieves state-of-the-art performances on several popular zero-shot skeleton-based action recognition benchmarks.
  arXiv  Detail & Related papers  (2024-09-22T06:44:58Z)
• A Simple Baseline with Single-encoder for Referring Image Segmentation [14.461024566536478]
  We present a novel RIS method with a single-encoder, i.e., BEiT-3, maximizing the potential of shared self-attention. Our simple baseline with a single encoder achieves outstanding performances on the RIS benchmark datasets.
  arXiv  Detail & Related papers  (2024-08-28T04:14:01Z)
• MacFormer: Semantic Segmentation with Fine Object Boundaries [38.430631361558426]
  We introduce a new semantic segmentation architecture, MacFormer'', which features two key components. Firstly, using learnable agent tokens, a Mutual Agent Cross-Attention (MACA) mechanism effectively facilitates the bidirectional integration of features across encoder and decoder layers. Secondly, a Frequency Enhancement Module (FEM) in the decoder leverages high-frequency and low-frequency components to boost features in the frequency domain. MacFormer is demonstrated to be compatible with various network architectures and outperforms existing methods in both accuracy and efficiency on datasets benchmark ADE20K and Cityscapes.
  arXiv  Detail & Related papers  (2024-08-11T05:36:10Z)
• Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion [57.232688209606515]
  We present HTCL, a novel Temporal Temporal Context Learning paradigm for improving camera-based semantic scene completion. Our method ranks $1st$ on the Semantic KITTI benchmark and even surpasses LiDAR-based methods in terms of mIoU.
  arXiv  Detail & Related papers  (2024-07-02T09:11:17Z)
• Sequence Shortening for Context-Aware Machine Translation [5.803309695504831]
  We show that a special case of multi-encoder architecture achieves higher accuracy on contrastive datasets. We introduce two novel methods - Latent Grouping and Latent Selecting, where the network learns to group tokens or selects the tokens to be cached as context.
  arXiv  Detail & Related papers  (2024-02-02T13:55:37Z)
• VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection [58.47940430618352]
  We propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) VadCLIP makes full use of fine-grained associations between vision and language on the strength of CLIP. We conduct extensive experiments on two commonly-used benchmarks, demonstrating that VadCLIP achieves the best performance on both coarse-grained and fine-grained WSVAD.
  arXiv  Detail & Related papers  (2023-08-22T14:58:36Z)
• DOAD: Decoupled One Stage Action Detection Network [77.14883592642782]
  Localizing people and recognizing their actions from videos is a challenging task towards high-level video understanding. Existing methods are mostly two-stage based, with one stage for person bounding box generation and the other stage for action recognition. We present a decoupled one-stage network dubbed DOAD, to improve the efficiency for-temporal action detection.
  arXiv  Detail & Related papers  (2023-04-01T08:06:43Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)
• SPTS v2: Single-Point Scene Text Spotting [146.98118405786445]
  New framework, SPTS v2, allows us to train high-performing text-spotting models using a single-point annotation. Tests show SPTS v2 can outperform previous state-of-the-art single-point text spotters with fewer parameters. Experiments suggest a potential preference for single-point representation in scene text spotting.
  arXiv  Detail & Related papers  (2023-01-04T14:20:14Z)
• Generalized Decoding for Pixel, Image, and Language [197.85760901840177]
  We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly. X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks.
  arXiv  Detail & Related papers  (2022-12-21T18:58:41Z)
• Joint Spatial-Temporal and Appearance Modeling with Transformer for Multiple Object Tracking [59.79252390626194]
  We propose a novel solution named TransSTAM, which leverages Transformer to model both the appearance features of each object and the spatial-temporal relationships among objects. The proposed method is evaluated on multiple public benchmarks including MOT16, MOT17, and MOT20, and it achieves a clear performance improvement in both IDF1 and HOTA.
  arXiv  Detail & Related papers  (2022-05-31T01:19:18Z)
• COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval [59.15034487974549]
  We propose a novel COllaborative Two-Stream vision-language pretraining model termed COTS for image-text retrieval. Our COTS achieves the highest performance among all two-stream methods and comparable performance with 10,800X faster in inference. Importantly, our COTS is also applicable to text-to-video retrieval, yielding new state-ofthe-art on the widely-used MSR-VTT dataset.
  arXiv  Detail & Related papers  (2022-04-15T12:34:47Z)
• Correlation-Aware Deep Tracking [83.51092789908677]
  We propose a novel target-dependent feature network inspired by the self-/cross-attention scheme. Our network deeply embeds cross-image feature correlation in multiple layers of the feature network. Our model can be flexibly pre-trained on abundant unpaired images, leading to notably faster convergence than the existing methods.
  arXiv  Detail & Related papers  (2022-03-03T11:53:54Z)
• End-to-End Dense Video Captioning with Parallel Decoding [53.34238344647624]
  We propose a simple yet effective framework for end-to-end dense video captioning with parallel decoding (PDVC) PDVC precisely segments the video into a number of event pieces under the holistic understanding of the video content. experiments on ActivityNet Captions and YouCook2 show that PDVC is capable of producing high-quality captioning results.
  arXiv  Detail & Related papers  (2021-08-17T17:39:15Z)
• Online Multiple Object Tracking with Cross-Task Synergy [120.70085565030628]
  We propose a novel unified model with synergy between position prediction and embedding association. The two tasks are linked by temporal-aware target attention and distractor attention, as well as identity-aware memory aggregation model.
  arXiv  Detail & Related papers  (2021-04-01T10:19:40Z)
• A Holistically-Guided Decoder for Deep Representation Learning with Applications to Semantic Segmentation and Object Detection [74.88284082187462]
  One common strategy is to adopt dilated convolutions in the backbone networks to extract high-resolution feature maps. We propose one novel holistically-guided decoder which is introduced to obtain the high-resolution semantic-rich feature maps.
  arXiv  Detail & Related papers  (2020-12-18T10:51:49Z)
• CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation [67.17625278621134]
  Video instance segmentation is a complex task in which we need to detect, segment, and track each object for any given video. Previous approaches only utilize single-frame features for the detection, segmentation, and tracking of objects. We propose a novel comprehensive feature aggregation approach (CompFeat) to refine features at both frame-level and object-level with temporal and spatial context information.
  arXiv  Detail & Related papers  (2020-12-07T00:31:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.