Awesome Multi-modal Object Tracking

• URL: http://arxiv.org/abs/2405.14200v2
• Date: Fri, 31 May 2024 11:09:59 GMT
• Title: Awesome Multi-modal Object Tracking
• Authors: Chunhui Zhang, Li Liu, Hao Wen, Xi Zhou, Yanfeng Wang,
• Abstract summary: Multi-modal object tracking (MMOT) is an emerging field that combines data from various modalities to estimate the state of an arbitrary object in a video sequence. To track the latest progress in MMOT, we conduct a comprehensive investigation in this report.
• Score: 41.76977058932557
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal object tracking (MMOT) is an emerging field that combines data from various modalities, \eg vision (RGB), depth, thermal infrared, event, language and audio, to estimate the state of an arbitrary object in a video sequence. It is of great significance for many applications such as autonomous driving and intelligent surveillance. In recent years, MMOT has received more and more attention. However, existing MMOT algorithms mainly focus on two modalities (\eg RGB+depth, RGB+thermal infrared, and RGB+language). To leverage more modalities, some recent efforts have been made to learn a unified visual object tracking model for any modality. Additionally, some large-scale multi-modal tracking benchmarks have been established by simultaneously providing more than two modalities, such as vision-language-audio (\eg WebUAV-3M) and vision-depth-language (\eg UniMod1K). To track the latest progress in MMOT, we conduct a comprehensive investigation in this report. Specifically, we first divide existing MMOT tasks into five main categories, \ie RGBL tracking, RGBE tracking, RGBD tracking, RGBT tracking, and miscellaneous (RGB+X), where X can be any modality, such as language, depth, and event. Then, we analyze and summarize each MMOT task, focusing on widely used datasets and mainstream tracking algorithms based on their technical paradigms (\eg self-supervised learning, prompt learning, knowledge distillation, generative models, and state space models). Finally, we maintain a continuously updated paper list for MMOT at https://github.com/983632847/Awesome-Multimodal-Object-Tracking.

Related papers

• Heterogeneous Graph Transformer for Multiple Tiny Object Tracking in RGB-T Videos [31.910202172609313]
  Existing multi-object tracking algorithms generally focus on single-modality scenes. We propose a novel framework called HGT-Track (Heterogeneous Graph Transformer based Multi-Tiny-Object Tracking) This paper introduces the first benchmark VT-Tiny-MOT (Visible-Thermal Tiny Multi-Object Tracking) for RGB-T fused multiple tiny object tracking.
  arXiv  Detail & Related papers  (2024-12-14T15:17:49Z)
• Visual Object Tracking across Diverse Data Modalities: A Review [33.006051781123595]
  Visual Object Tracking (VOT) is an attractive and significant research area in computer vision. We first review three types of mainstream single-modal VOT, including RGB, thermal infrared and point cloud tracking. Then we summarize four kinds of multi-modal VOT, including RGB-Depth, RGB-Thermal, RGB-LiDAR and RGB-Language.
  arXiv  Detail & Related papers  (2024-12-13T09:25:18Z)
• Single-Model and Any-Modality for Video Object Tracking [85.83753760853142]
  We introduce Un-Track, a Unified Tracker of a single set of parameters for any modality. To handle any modality, our method learns their common latent space through low-rank factorization and reconstruction techniques. Our Un-Track achieves +8.1 absolute F-score gain, on the DepthTrack dataset, by introducing only +2.14 (over 21.50) GFLOPs with +6.6M (over 93M) parameters.
  arXiv  Detail & Related papers  (2023-11-27T14:17:41Z)
• ReIDTracker Sea: the technical report of BoaTrack and SeaDronesSee-MOT challenge at MaCVi of WACV24 [0.0]
  Our solution tries to explore Multi-Object Tracking in maritime Unmanned Aerial vehicles (UAVs) and Unmanned Surface Vehicles (USVs) usage scenarios. The scheme achieved top 3 performance on both UAV-based Multi-Object Tracking with Reidentification and USV-based Multi-Object Tracking benchmarks.
  arXiv  Detail & Related papers  (2023-11-12T07:37:07Z)
• OVTrack: Open-Vocabulary Multiple Object Tracking [64.73379741435255]
  OVTrack is an open-vocabulary tracker capable of tracking arbitrary object classes. It sets a new state-of-the-art on the large-scale, large-vocabulary TAO benchmark.
  arXiv  Detail & Related papers  (2023-04-17T16:20:05Z)
• Visual Prompt Multi-Modal Tracking [71.53972967568251]
  Visual Prompt multi-modal Tracking (ViPT) learns the modal-relevant prompts to adapt the frozen pre-trained foundation model to various downstream multimodal tracking tasks. ViPT outperforms the full fine-tuning paradigm on multiple downstream tracking tasks including RGB+Depth, RGB+Thermal, and RGB+Event tracking.
  arXiv  Detail & Related papers  (2023-03-20T01:51:07Z)
• Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline [80.13652104204691]
  In this paper, we construct a large-scale benchmark with high diversity for visible-thermal UAV tracking (VTUAV) We provide a coarse-to-fine attribute annotation, where frame-level attributes are provided to exploit the potential of challenge-specific trackers. In addition, we design a new RGB-T baseline, named Hierarchical Multi-modal Fusion Tracker (HMFT), which fuses RGB-T data in various levels.
  arXiv  Detail & Related papers  (2022-04-08T15:22:33Z)
• Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Driving [22.693895321632507]
  We propose a probabilistic, multi-modal, multi-object tracking system consisting of different trainable modules. We show that our method outperforms current state-of-the-art on the NuScenes Tracking dataset.
  arXiv  Detail & Related papers  (2020-12-26T15:00:54Z)
• TAO: A Large-Scale Benchmark for Tracking Any Object [95.87310116010185]
  Tracking Any Object dataset consists of 2,907 high resolution videos, captured in diverse environments, which are half a minute long on average. We ask annotators to label objects that move at any point in the video, and give names to them post factum. Our vocabulary is both significantly larger and qualitatively different from existing tracking datasets.
  arXiv  Detail & Related papers  (2020-05-20T21:07:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.