ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Tracking

• URL: http://arxiv.org/abs/2505.20381v1
• Date: Mon, 26 May 2025 17:55:19 GMT
• Title: ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Tracking
• Authors: Sijia Chen, Yanqiu Yu, En Yu, Wenbing Tao,
• Abstract summary: We propose a new task, called Reasoning-based Multi-Object Tracking (ReaMOT)<n>ReaMOT is a more challenging task that requires accurate reasoning about objects that match the language instruction with reasoning characteristic and tracking the objects' trajectories.<n>We construct ReaMOT Challenge, a reasoning-based multi-object tracking benchmark built upon 12 datasets.
• Score: 23.76697700853566
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Referring Multi-object tracking (RMOT) is an important research field in computer vision. Its task form is to guide the models to track the objects that conform to the language instruction. However, the RMOT task commonly requires clear language instructions, such methods often fail to work when complex language instructions with reasoning characteristics appear. In this work, we propose a new task, called Reasoning-based Multi-Object Tracking (ReaMOT). ReaMOT is a more challenging task that requires accurate reasoning about objects that match the language instruction with reasoning characteristic and tracking the objects' trajectories. To advance the ReaMOT task and evaluate the reasoning capabilities of tracking models, we construct ReaMOT Challenge, a reasoning-based multi-object tracking benchmark built upon 12 datasets. Specifically, it comprises 1,156 language instructions with reasoning characteristic, 423,359 image-language pairs, and 869 diverse scenes, which is divided into three levels of reasoning difficulty. In addition, we propose a set of evaluation metrics tailored for the ReaMOT task. Furthermore, we propose ReaTrack, a training-free framework for reasoning-based multi-object tracking based on large vision-language models (LVLM) and SAM2, as a baseline for the ReaMOT task. Extensive experiments on the ReaMOT Challenge benchmark demonstrate the effectiveness of our ReaTrack framework.

Related papers

• Analyze-Prompt-Reason: A Collaborative Agent-Based Framework for Multi-Image Vision-Language Reasoning [3.588567067449924]
  We present a Collaborative Agent-Based Framework for Multi-Image Reasoning.<n>Our approach tackles the challenge of interleaved multimodal reasoning across diverse datasets and task formats.<n>We evaluate our method on 18 diverse datasets from the 2025 MIRAGE Challenge.
  arXiv  Detail & Related papers  (2025-08-01T06:39:15Z)
• LaMOT: Language-Guided Multi-Object Tracking [13.866428951384124]
  Vision-Language MOT aims to track objects based on human language commands. Despite various efforts, a key challenge lies in the lack of a clear understanding of why language is used for tracking. We introduce Language-Guided MOT, a unified task framework, along with a corresponding large-scale benchmark, termed LaMOT.
  arXiv  Detail & Related papers  (2024-06-12T15:24:09Z)
• Single-Shot and Multi-Shot Feature Learning for Multi-Object Tracking [55.13878429987136]
  We propose a simple yet effective two-stage feature learning paradigm to jointly learn single-shot and multi-shot features for different targets. Our method has achieved significant improvements on MOT17 and MOT20 datasets while reaching state-of-the-art performance on DanceTrack dataset.
  arXiv  Detail & Related papers  (2023-11-17T08:17:49Z)
• SimpleMTOD: A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representation [2.4469484645516837]
  SimpleMTOD recasts several sub-tasks in multimodal task-oriented dialogues as sequence prediction tasks. We introduce both local and de-localized tokens for objects within a scene. The model does not rely on task-specific architectural changes such as classification heads.
  arXiv  Detail & Related papers  (2023-07-10T21:16:46Z)
• Referring Multi-Object Tracking [78.63827591797124]
  We propose a new and general referring understanding task, termed referring multi-object tracking (RMOT) Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking. To the best of our knowledge, it is the first work to achieve an arbitrary number of referent object predictions in videos.
  arXiv  Detail & Related papers  (2023-03-06T18:50:06Z)
• Unifying Tracking and Image-Video Object Detection [54.91658924277527]
  TrIVD (Tracking and Image-Video Detection) is the first framework that unifies image OD, video OD, and MOT within one end-to-end model. To handle the discrepancies and semantic overlaps of category labels, TrIVD formulates detection/tracking as grounding and reasons about object categories.
  arXiv  Detail & Related papers  (2022-11-20T20:30:28Z)
• End-to-end Tracking with a Multi-query Transformer [96.13468602635082]
  Multiple-object tracking (MOT) is a challenging task that requires simultaneous reasoning about location, appearance, and identity of the objects in the scene over time. Our aim in this paper is to move beyond tracking-by-detection approaches, to class-agnostic tracking that performs well also for unknown object classes.
  arXiv  Detail & Related papers  (2022-10-26T10:19:37Z)
• Unified Transformer Tracker for Object Tracking [58.65901124158068]
  We present the Unified Transformer Tracker (UTT) to address tracking problems in different scenarios with one paradigm. A track transformer is developed in our UTT to track the target in both Single Object Tracking (SOT) and Multiple Object Tracking (MOT)
  arXiv  Detail & Related papers  (2022-03-29T01:38:49Z)
• Multi-target tracking for video surveillance using deep affinity network: a brief review [0.0]
  Multi-target tracking (MTT) for video surveillance is one of the important and challenging tasks. Deep learning models are known to function like the human brain.
  arXiv  Detail & Related papers  (2021-10-29T10:44:26Z)
• MOPT: Multi-Object Panoptic Tracking [33.77171216778909]
  We introduce a novel perception task denoted as multi-object panoptic tracking (MOPT) MOPT allows for exploiting pixel-level semantic information of 'thing' and'stuff' classes, temporal coherence, and pixel-level associations over time. We present extensive quantitative and qualitative evaluations of both vision-based and LiDAR-based MOPT that demonstrate encouraging results.
  arXiv  Detail & Related papers  (2020-04-17T11:45:28Z)
• A Unified Object Motion and Affinity Model for Online Multi-Object Tracking [127.5229859255719]
  We propose a novel MOT framework that unifies object motion and affinity model into a single network, named UMA. UMA integrates single object tracking and metric learning into a unified triplet network by means of multi-task learning. We equip our model with a task-specific attention module, which is used to boost task-aware feature learning.
  arXiv  Detail & Related papers  (2020-03-25T09:36:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.