LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation

• URL: http://arxiv.org/abs/2404.19326v2
• Date: Wed, 1 May 2024 01:30:58 GMT
• Title: LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation
• Authors: Lingyi Hong, Zhongying Liu, Wenchao Chen, Chenzhi Tan, Yuang Feng, Xinyu Zhou, Pinxue Guo, Jinglun Li, Zhaoyu Chen, Shuyong Gao, Wei Zhang, Wenqiang Zhang,
• Abstract summary: Video object segmentation (VOS) aims to distinguish and track target objects in a video. Existing benchmarks mainly focus on short-term videos, where objects remain visible most of the time. We propose a novel benchmark named LVOS, comprising 720 videos with 296,401 frames and 407,945 high-quality annotations. Videos in LVOS last 1.14 minutes on average, approximately 5 times longer than videos in existing datasets.
• Score: 29.07092353094942
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video object segmentation (VOS) aims to distinguish and track target objects in a video. Despite the excellent performance achieved by off-the-shell VOS models, existing VOS benchmarks mainly focus on short-term videos lasting about 5 seconds, where objects remain visible most of the time. However, these benchmarks poorly represent practical applications, and the absence of long-term datasets restricts further investigation of VOS in realistic scenarios. Thus, we propose a novel benchmark named LVOS, comprising 720 videos with 296,401 frames and 407,945 high-quality annotations. Videos in LVOS last 1.14 minutes on average, approximately 5 times longer than videos in existing datasets. Each video includes various attributes, especially challenges deriving from the wild, such as long-term reappearing and cross-temporal similar objects. Compared to previous benchmarks, our LVOS better reflects VOS models' performance in real scenarios. Based on LVOS, we evaluate 20 existing VOS models under 4 different settings and conduct a comprehensive analysis. On LVOS, these models suffer a large performance drop, highlighting the challenge of achieving precise tracking and segmentation in real-world scenarios. Attribute-based analysis indicates that key factor to accuracy decline is the increased video length, emphasizing LVOS's crucial role. We hope our LVOS can advance development of VOS in real scenes. Data and code are available at https://lingyihongfd.github.io/lvos.github.io/.

Related papers

• Discriminative Spatial-Semantic VOS Solution: 1st Place Solution for 6th LSVOS [68.47681139026666]
  Video object segmentation (VOS) is a crucial task in computer vision. Current VOS methods struggle with complex scenes and prolonged object motions. This report introduces a discriminative spatial-temporal VOS model.
  arXiv  Detail & Related papers  (2024-08-29T10:47:17Z)
• Efficient Video Object Segmentation via Modulated Cross-Attention Memory [123.12273176475863]
  We propose a transformer-based approach, named MAVOS, to model temporal smoothness without requiring frequent memory expansion. Our MAVOS achieves a J&F score of 63.3% while operating at 37 frames per second (FPS) on a single V100 GPU.
  arXiv  Detail & Related papers  (2024-03-26T17:59:58Z)
• Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples [61.66967790884943]
  Referring video object segmentation (RVOS) relies on sufficient data for a given scene. In more realistic scenarios, only minimal annotations are available for a new scene. We propose a model with a newly designed cross-modal affinity (CMA) module based on a Transformer architecture. CMA module builds multimodal affinity with a few samples, thus quickly learning new semantic information, and enabling the model to adapt to different scenarios.
  arXiv  Detail & Related papers  (2023-09-05T08:34:23Z)
• MOSE: A New Dataset for Video Object Segmentation in Complex Scenes [106.64327718262764]
  Video object segmentation (VOS) aims at segmenting a particular object throughout the entire video clip sequence. The state-of-the-art VOS methods have achieved excellent performance (e.g., 90+% J&F) on existing datasets. We collect a new VOS dataset called coMplex video Object SEgmentation (MOSE) to study the tracking and segmenting objects in complex environments.
  arXiv  Detail & Related papers  (2023-02-03T17:20:03Z)
• LVOS: A Benchmark for Long-term Video Object Segmentation [31.76468328063721]
  We present a new benchmark dataset named textbfLVOS, which consists of 220 videos with a total duration of 421 minutes. The videos in our LVOS last 1.59 minutes on average, which is 20 times longer than videos in existing VOS datasets. We propose a Diverse Dynamic Memory network (DDMemory) that consists of three complementary memory banks to exploit temporal information adequately.
  arXiv  Detail & Related papers  (2022-11-18T11:59:37Z)
• Region Aware Video Object Segmentation with Deep Motion Modeling [56.95836951559529]
  Region Aware Video Object (RAVOS) is a method that predicts regions of interest for efficient object segmentation and memory storage. For efficient segmentation, object features are extracted according to the ROIs, and an object decoder is designed for object-level segmentation. For efficient memory storage, we propose motion path memory to filter out redundant context by memorizing the features within the motion path of objects between two frames.
  arXiv  Detail & Related papers  (2022-07-21T01:44:40Z)
• 5th Place Solution for YouTube-VOS Challenge 2022: Video Object Segmentation [4.004851693068654]
  Video object segmentation (VOS) has made significant progress with the rise of deep learning. Similar objects are easily confused and tiny objects are difficult to find. We propose a simple yet effective solution for this task.
  arXiv  Detail & Related papers  (2022-06-20T06:14:27Z)
• Learning Video Object Segmentation from Unlabeled Videos [158.18207922363783]
  We propose a new method for video object segmentation (VOS) that addresses object pattern learning from unlabeled videos. We introduce a unified unsupervised/weakly supervised learning framework, called MuG, that comprehensively captures properties of VOS at multiple granularities.
  arXiv  Detail & Related papers  (2020-03-10T22:12:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.