360VOTS: Visual Object Tracking and Segmentation in Omnidirectional Videos

• URL: http://arxiv.org/abs/2404.13953v1
• Date: Mon, 22 Apr 2024 07:54:53 GMT
• Title: 360VOTS: Visual Object Tracking and Segmentation in Omnidirectional Videos
• Authors: Yinzhe Xu, Huajian Huang, Yingshu Chen, Sai-Kit Yeung,
• Abstract summary: We propose a comprehensive dataset and benchmark that incorporates a new component called omnidirectional video object segmentation (360VOS) 360VOS dataset includes 290 sequences accompanied by dense pixel-wise masks and covers a broader range of target categories. We benchmark state-of-the-art approaches and demonstrate the effectiveness of our proposed 360 tracking framework and training dataset.
• Score: 16.372814014632944
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual object tracking and segmentation in omnidirectional videos are challenging due to the wide field-of-view and large spherical distortion brought by 360{\deg} images. To alleviate these problems, we introduce a novel representation, extended bounding field-of-view (eBFoV), for target localization and use it as the foundation of a general 360 tracking framework which is applicable for both omnidirectional visual object tracking and segmentation tasks. Building upon our previous work on omnidirectional visual object tracking (360VOT), we propose a comprehensive dataset and benchmark that incorporates a new component called omnidirectional video object segmentation (360VOS). The 360VOS dataset includes 290 sequences accompanied by dense pixel-wise masks and covers a broader range of target categories. To support both the development and evaluation of algorithms in this domain, we divide the dataset into a training subset with 170 sequences and a testing subset with 120 sequences. Furthermore, we tailor evaluation metrics for both omnidirectional tracking and segmentation to ensure rigorous assessment. Through extensive experiments, we benchmark state-of-the-art approaches and demonstrate the effectiveness of our proposed 360 tracking framework and training dataset. Homepage: https://360vots.hkustvgd.com/

Related papers

• 3D-Aware Instance Segmentation and Tracking in Egocentric Videos [107.10661490652822]
  Egocentric videos present unique challenges for 3D scene understanding. This paper introduces a novel approach to instance segmentation and tracking in first-person video. By incorporating spatial and temporal cues, we achieve superior performance compared to state-of-the-art 2D approaches.
  arXiv  Detail & Related papers  (2024-08-19T10:08:25Z)
• 360VOT: A New Benchmark Dataset for Omnidirectional Visual Object Tracking [10.87309734945868]
  360deg images can provide an omnidirectional field of view which is important for stable and long-term scene perception. In this paper, we explore 360deg images for visual object tracking and perceive new challenges caused by large distortion. We propose a new large-scale omnidirectional tracking benchmark dataset, 360VOT, in order to facilitate future research.
  arXiv  Detail & Related papers  (2023-07-27T05:32:01Z)
• Segment Anything Meets Point Tracking [116.44931239508578]
  This paper presents a novel method for point-centric interactive video segmentation, empowered by SAM and long-term point tracking. We highlight the merits of point-based tracking through direct evaluation on the zero-shot open-world Unidentified Video Objects (UVO) benchmark. Our experiments on popular video object segmentation and multi-object segmentation tracking benchmarks, including DAVIS, YouTube-VOS, and BDD100K, suggest that a point-based segmentation tracker yields better zero-shot performance and efficient interactions.
  arXiv  Detail & Related papers  (2023-07-03T17:58:01Z)
• DIVOTrack: A Novel Dataset and Baseline Method for Cross-View Multi-Object Tracking in DIVerse Open Scenes [74.64897845999677]
  We introduce a new cross-view multi-object tracking dataset for DIVerse Open scenes with dense tracking pedestrians. Our DIVOTrack has fifteen distinct scenarios and 953 cross-view tracks, surpassing all cross-view multi-object tracking datasets currently available. Furthermore, we provide a novel baseline cross-view tracking method with a unified joint detection and cross-view tracking framework named CrossMOT.
  arXiv  Detail & Related papers  (2023-02-15T14:10:42Z)
• CXTrack: Improving 3D Point Cloud Tracking with Contextual Information [59.55870742072618]
  3D single object tracking plays an essential role in many applications, such as autonomous driving. We propose CXTrack, a novel transformer-based network for 3D object tracking. We show that CXTrack achieves state-of-the-art tracking performance while running at 29 FPS.
  arXiv  Detail & Related papers  (2022-11-12T11:29:01Z)
• Field-of-View IoU for Object Detection in 360{\deg} Images [36.72543749626039]
  We propose two fundamental techniques -- Field-of-View IoU (FoV-IoU) and 360Augmentation for object detection in 360deg images. FoV-IoU computes the intersection-over-union of two Field-of-View bounding boxes in a spherical image which could be used for training, inference, and evaluation. 360Augmentation is a data augmentation technique specific to 360deg object detection task which randomly rotates a spherical image and solves the bias due to the sphere-to-plane projection.
  arXiv  Detail & Related papers  (2022-02-07T14:01:59Z)
• Capturing Omni-Range Context for Omnidirectional Segmentation [29.738065412097598]
  We introduce Concurrent Attention Networks (ECANets) to bridge the gap in terms of FoV and structural distribution between the imaging domains. We upgrade model training by leveraging multi-source and omni-supervised learning, taking advantage of both: Densely labeled and unlabeled data. Our novel model, training regimen and multisource prediction fusion elevate the performance (mIoU) to new state-of-the-art results.
  arXiv  Detail & Related papers  (2021-03-09T19:46:09Z)
• Visual Tracking by TridentAlign and Context Embedding [71.60159881028432]
  We propose novel TridentAlign and context embedding modules for Siamese network-based visual tracking methods. The performance of the proposed tracker is comparable to that of state-of-the-art trackers, while the proposed tracker runs at real-time speed.
  arXiv  Detail & Related papers  (2020-07-14T08:00:26Z)
• TAO: A Large-Scale Benchmark for Tracking Any Object [95.87310116010185]
  Tracking Any Object dataset consists of 2,907 high resolution videos, captured in diverse environments, which are half a minute long on average. We ask annotators to label objects that move at any point in the video, and give names to them post factum. Our vocabulary is both significantly larger and qualitatively different from existing tracking datasets.
  arXiv  Detail & Related papers  (2020-05-20T21:07:28Z)
• A Fixation-based 360{\deg} Benchmark Dataset for Salient Object Detection [21.314578493964333]
  Fixation prediction (FP) in panoramic contents has been widely investigated along with the booming trend of virtual reality (VR) applications. salient object detection (SOD) has been seldom explored in 360deg images due to the lack of datasets representative of real scenes.
  arXiv  Detail & Related papers  (2020-01-22T11:16:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.