Just Functioning as a Hook for Two-Stage Referring Multi-Object Tracking

• URL: http://arxiv.org/abs/2503.07516v2
• Date: Fri, 16 May 2025 02:34:02 GMT
• Title: Just Functioning as a Hook for Two-Stage Referring Multi-Object Tracking
• Authors: Weize Li, Yunhao Du, Qixiang Yin, Zhicheng Zhao, Fei Su, Daqi Liu,
• Abstract summary: Referring Multi-Object Tracking (RMOT) aims to localize target trajectories specified by natural language expressions in videos.<n>We propose JustHook, a novel hook-like framework for two-stage RMOT.
• Score: 22.669740476582835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring Multi-Object Tracking (RMOT) aims to localize target trajectories specified by natural language expressions in videos. Existing RMOT methods mainly follow two paradigms: one-stage strategies and two-stage ones. The former jointly trains tracking with referring but suffers from substantial computational overhead. Although the latter improves efficiency, it overlooks the inherent contextual aggregation capabilities of pre-trained visual backbones and takes a detour. Meanwhile, its fixed dual-tower architecture restricts compatibility with other visual / text backbones. To address these limitations, we propose JustHook, a novel hook-like framework for two-stage RMOT, which introduces two core components: (1) a Visual Feature Hook (VFH), enabling JustHook to extract context-rich local features directly from the original visual backbone like a hook; (2) a Parallel Combined Decoder (PCD), which transforms the passive cosine similarity measurement between independent modalities into active contrastive learning within the combined feature space. The proposed JustHook not only leverages the capabilities of pre-trained models but also breaks free from the constraints of inherent modality alignment, achieving strong scalability. Extensive experiments on Refer-KITTI and Refer-KITTI-V2 demonstrate that JustHook outperforms state-of-the-art methods across diverse encoder combinations, achieving a notable 7.77\% HOTA improvement on Refer-KITTI-V2. Code will be made available soon.

Related papers

• Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition [36.662223760818584]
  Trokens is a novel approach that transforms trajectory points into semantic-aware relational tokens for action recognition.<n>We develop a motion modeling framework that captures both intra-trajectory dynamics through the Histogram of Oriented Displacements (HoD) and inter-trajectory relationships to model complex action patterns.<n>Our approach effectively combines these trajectory tokens with semantic features to enhance appearance features with motion information, achieving state-of-the-art performance across six diverse few-shot action recognition benchmarks.
  arXiv  Detail & Related papers  (2025-08-05T17:59:58Z)
• DINOv2-powered Few-Shot Semantic Segmentation: A Unified Framework via Cross-Model Distillation and 4D Correlation Mining [30.564216896513596]
  Few-shot semantic segmentation has gained increasing interest due to its generalization capability.<n>Recent approaches have turned to foundation models to enhance representation transferability.<n>We propose FS-DINO, with only DINOv2's encoder and a lightweight segmenter.
  arXiv  Detail & Related papers  (2025-04-22T07:47:06Z)
• Cognitive Disentanglement for Referring Multi-Object Tracking [28.325814292139686]
  We propose a Cognitive Disentanglement for Referring Multi-Object Tracking (CDRMT) framework. CDRMT adapts the "what" and "where" pathways from the human visual processing system to RMOT tasks. Experiments on different benchmark datasets demonstrate that CDRMT achieves substantial improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-03-14T15:21:54Z)
• Col-OLHTR: A Novel Framework for Multimodal Online Handwritten Text Recognition [82.88856416080331]
  Online Handwritten Text Recognition (OLHTR) has gained considerable attention for its diverse range of applications.<n>Current approaches usually treat OLHTR as a sequence recognition task, employing either a single trajectory or image encoder, or multi-stream encoders.<n>We propose a Collaborative learning-based OLHTR framework, called Col-OLHTR, that learns multimodal features during training while maintaining a single-stream inference process.
  arXiv  Detail & Related papers  (2025-02-10T02:12:24Z)
• Zero-Shot Skeleton-based Action Recognition with Dual Visual-Text Alignment [11.72557768532557]
  Key to zero-shot action recognition lies in aligning visual features with semantic vectors representing action categories. Our approach achieves state-of-the-art performances on several popular zero-shot skeleton-based action recognition benchmarks.
  arXiv  Detail & Related papers  (2024-09-22T06:44:58Z)
• A Simple Baseline with Single-encoder for Referring Image Segmentation [14.461024566536478]
  We present a novel RIS method with a single-encoder, i.e., BEiT-3, maximizing the potential of shared self-attention. Our simple baseline with a single encoder achieves outstanding performances on the RIS benchmark datasets.
  arXiv  Detail & Related papers  (2024-08-28T04:14:01Z)
• MacFormer: Semantic Segmentation with Fine Object Boundaries [38.430631361558426]
  We introduce a new semantic segmentation architecture, MacFormer'', which features two key components. Firstly, using learnable agent tokens, a Mutual Agent Cross-Attention (MACA) mechanism effectively facilitates the bidirectional integration of features across encoder and decoder layers. Secondly, a Frequency Enhancement Module (FEM) in the decoder leverages high-frequency and low-frequency components to boost features in the frequency domain. MacFormer is demonstrated to be compatible with various network architectures and outperforms existing methods in both accuracy and efficiency on datasets benchmark ADE20K and Cityscapes.
  arXiv  Detail & Related papers  (2024-08-11T05:36:10Z)
• Sequence Shortening for Context-Aware Machine Translation [5.803309695504831]
  We show that a special case of multi-encoder architecture achieves higher accuracy on contrastive datasets. We introduce two novel methods - Latent Grouping and Latent Selecting, where the network learns to group tokens or selects the tokens to be cached as context.
  arXiv  Detail & Related papers  (2024-02-02T13:55:37Z)
• VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection [58.47940430618352]
  We propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) VadCLIP makes full use of fine-grained associations between vision and language on the strength of CLIP. We conduct extensive experiments on two commonly-used benchmarks, demonstrating that VadCLIP achieves the best performance on both coarse-grained and fine-grained WSVAD.
  arXiv  Detail & Related papers  (2023-08-22T14:58:36Z)
• DOAD: Decoupled One Stage Action Detection Network [77.14883592642782]
  Localizing people and recognizing their actions from videos is a challenging task towards high-level video understanding. Existing methods are mostly two-stage based, with one stage for person bounding box generation and the other stage for action recognition. We present a decoupled one-stage network dubbed DOAD, to improve the efficiency for-temporal action detection.
  arXiv  Detail & Related papers  (2023-04-01T08:06:43Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)
• SPTS v2: Single-Point Scene Text Spotting [146.98118405786445]
  New framework, SPTS v2, allows us to train high-performing text-spotting models using a single-point annotation. Tests show SPTS v2 can outperform previous state-of-the-art single-point text spotters with fewer parameters. Experiments suggest a potential preference for single-point representation in scene text spotting.
  arXiv  Detail & Related papers  (2023-01-04T14:20:14Z)
• Generalized Decoding for Pixel, Image, and Language [197.85760901840177]
  We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly. X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks.
  arXiv  Detail & Related papers  (2022-12-21T18:58:41Z)
• Joint Spatial-Temporal and Appearance Modeling with Transformer for Multiple Object Tracking [59.79252390626194]
  We propose a novel solution named TransSTAM, which leverages Transformer to model both the appearance features of each object and the spatial-temporal relationships among objects. The proposed method is evaluated on multiple public benchmarks including MOT16, MOT17, and MOT20, and it achieves a clear performance improvement in both IDF1 and HOTA.
  arXiv  Detail & Related papers  (2022-05-31T01:19:18Z)
• COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval [59.15034487974549]
  We propose a novel COllaborative Two-Stream vision-language pretraining model termed COTS for image-text retrieval. Our COTS achieves the highest performance among all two-stream methods and comparable performance with 10,800X faster in inference. Importantly, our COTS is also applicable to text-to-video retrieval, yielding new state-ofthe-art on the widely-used MSR-VTT dataset.
  arXiv  Detail & Related papers  (2022-04-15T12:34:47Z)
• Correlation-Aware Deep Tracking [83.51092789908677]
  We propose a novel target-dependent feature network inspired by the self-/cross-attention scheme. Our network deeply embeds cross-image feature correlation in multiple layers of the feature network. Our model can be flexibly pre-trained on abundant unpaired images, leading to notably faster convergence than the existing methods.
  arXiv  Detail & Related papers  (2022-03-03T11:53:54Z)
• End-to-End Dense Video Captioning with Parallel Decoding [53.34238344647624]
  We propose a simple yet effective framework for end-to-end dense video captioning with parallel decoding (PDVC) PDVC precisely segments the video into a number of event pieces under the holistic understanding of the video content. experiments on ActivityNet Captions and YouCook2 show that PDVC is capable of producing high-quality captioning results.
  arXiv  Detail & Related papers  (2021-08-17T17:39:15Z)
• A Holistically-Guided Decoder for Deep Representation Learning with Applications to Semantic Segmentation and Object Detection [74.88284082187462]
  One common strategy is to adopt dilated convolutions in the backbone networks to extract high-resolution feature maps. We propose one novel holistically-guided decoder which is introduced to obtain the high-resolution semantic-rich feature maps.
  arXiv  Detail & Related papers  (2020-12-18T10:51:49Z)
• CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation [67.17625278621134]
  Video instance segmentation is a complex task in which we need to detect, segment, and track each object for any given video. Previous approaches only utilize single-frame features for the detection, segmentation, and tracking of objects. We propose a novel comprehensive feature aggregation approach (CompFeat) to refine features at both frame-level and object-level with temporal and spatial context information.
  arXiv  Detail & Related papers  (2020-12-07T00:31:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.