MSC: A Marine Wildlife Video Dataset with Grounded Segmentation and Clip-Level Captioning

• URL: http://arxiv.org/abs/2508.04549v1
• Date: Wed, 06 Aug 2025 15:34:24 GMT
• Title: MSC: A Marine Wildlife Video Dataset with Grounded Segmentation and Clip-Level Captioning
• Abstract summary: Marine videos present significant challenges for video understanding.<n>Existing video captioning datasets often fail to generalize to the complexities of the marine environment.<n>We propose a two-stage marine object-oriented video captioning pipeline.
• Score: 15.968772405167877
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Marine videos present significant challenges for video understanding due to the dynamics of marine objects and the surrounding environment, camera motion, and the complexity of underwater scenes. Existing video captioning datasets, typically focused on generic or human-centric domains, often fail to generalize to the complexities of the marine environment and gain insights about marine life. To address these limitations, we propose a two-stage marine object-oriented video captioning pipeline. We introduce a comprehensive video understanding benchmark that leverages the triplets of video, text, and segmentation masks to facilitate visual grounding and captioning, leading to improved marine video understanding and analysis, and marine video generation. Additionally, we highlight the effectiveness of video splitting in order to detect salient object transitions in scene changes, which significantly enrich the semantics of captioning content. Our dataset and code have been released at https://msc.hkustvgd.com.

Related papers

• MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning [21.267648894621683]
  We focus on marine video understanding, which requires substantial domain expertise and large-scale annotated video data.<n>To address these challenges, we carefully curate the first event-centric marine video understanding dataset called MarineEVT.<n>Based on MarineEVT, we decompose marine video understanding as an Event-centric Visual Tool-integrated Reasoning process EVT-R1.
  arXiv  Detail & Related papers  (2026-07-27T07:06:14Z)
• Video4Spatial: Towards Visuospatial Intelligence with Context-Guided Video Generation [32.94087966263353]
  Video generative models conditioned solely on video-based scene context can perform complex spatial tasks.<n>We present Video4Spatial, a framework showing that video diffusion models conditioned solely on video-based scene context can perform complex spatial tasks.
  arXiv  Detail & Related papers  (2025-12-02T18:59:44Z)
• HieraMamba: Video Temporal Grounding via Hierarchical Anchor-Mamba Pooling [52.10845971383909]
  HieraMamba is a hierarchical architecture that preserves temporal structure and semantic richness across scales.<n>It sets a new state-of-the-art on Ego4D-NLQ, MAD, and TACoS, demonstrating precise, temporally faithful localization in long, untrimmed videos.
  arXiv  Detail & Related papers  (2025-10-27T06:13:07Z)
• Expose Camouflage in the Water: Underwater Camouflaged Instance Segmentation and Dataset [76.92197418745822]
  camouflaged instance segmentation (CIS) faces greater challenges in accurately segmenting objects that blend closely with their surroundings.<n>Traditional camouflaged instance segmentation methods, trained on terrestrial-dominated datasets with limited underwater samples, may exhibit inadequate performance in underwater scenes.<n>We introduce the first underwater camouflaged instance segmentation dataset, UCIS4K, which comprises 3,953 images of camouflaged marine organisms with instance-level annotations.
  arXiv  Detail & Related papers  (2025-10-20T14:34:51Z)
• SpatialVID: A Large-Scale Video Dataset with Spatial Annotations [58.01259302233675]
  SpatialVID is a dataset of in-the-wild videos with diverse scenes, camera movements and dense 3D annotations such as per-frame camera poses, depth, and motion instructions.<n>We collect more than 21,000 hours of raw video, and process them into 2.7 million clips through a hierarchical filtering pipeline.<n>A subsequent annotation pipeline enriches these clips with detailed spatial and semantic information, including camera poses, depth maps, dynamic masks, structured captions, and serialized motion instructions.
  arXiv  Detail & Related papers  (2025-09-11T17:59:31Z)
• VoCap: Video Object Captioning and Segmentation from Any Prompt [78.90048335805047]
  VoCap is a flexible model that consumes a video segmentation and a prompt understanding of various modalities.<n>It addresses promptable video object segmentation, referring, and object captioning.<n>Our model yields state-the-art results on referring expression video object segmentation.
  arXiv  Detail & Related papers  (2025-08-29T17:43:58Z)
• Controllable Hybrid Captioner for Improved Long-form Video Understanding [0.24578723416255746]
  Video data is extremely dense and high-dimensional.<n>Text-based summaries of video content offer a way to represent content in a much more compact manner than raw.<n>We introduce Vision Language Models (VLMs) to enrich the memory with static scene descriptions.
  arXiv  Detail & Related papers  (2025-07-22T22:09:00Z)
• SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models [80.3895950009792]
  Achieving fine-grained-temporal understanding in videos remains a major challenge for current Video Large Multimodels (Video LMMs)<n>We contribute in three core aspects: dataset, model, and benchmark.<n>First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically to enable joint learning of video understanding, grounding, and multi-turn video chat.<n>Second, we propose the SAMA model, which incorporates a versatile-temporal context aggregator and a Segment Model to jointly enhance fine-grained video comprehension and precise grounding capabilities.
  arXiv  Detail & Related papers  (2025-05-24T18:13:16Z)
• SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning [50.98341607245458]
  Masked video modeling is an effective paradigm for video self-supervised learning (SSL)<n>This paper introduces a novel SSL approach for video representation learning, dubbed as SMILE, by infusing both spatial and motion semantics.<n>We establish a new self-supervised video learning paradigm capable of learning strong video representations without requiring any natural video data.
  arXiv  Detail & Related papers  (2025-04-01T08:20:55Z)
• Segment Any Motion in Videos [80.72424676419755]
  We propose a novel approach for moving object segmentation that combines long-range trajectory motion cues with DINO-based semantic features.<n>Our model employs Spatio-Temporal Trajectory Attention and Motion-Semantic Decoupled Embedding to prioritize motion while integrating semantic support.
  arXiv  Detail & Related papers  (2025-03-28T09:34:11Z)
• AUTV: Creating Underwater Video Datasets with Pixel-wise Annotations [27.609227883183713]
  We propose AUTV, a framework for synthesizing marine video data with pixel-wise annotations.<n>We demonstrate the effectiveness of this framework by constructing two video datasets.
  arXiv  Detail & Related papers  (2025-03-17T05:18:20Z)
• OmniVid: A Generative Framework for Universal Video Understanding [133.73878582161387]
  We seek to unify the output space of video understanding tasks by using languages as labels and additionally introducing time and box tokens. This enables us to address various types of video tasks, including classification, captioning, and localization. We demonstrate such a simple and straightforward idea is quite effective and can achieve state-of-the-art or competitive results.
  arXiv  Detail & Related papers  (2024-03-26T17:59:24Z)
• MarineVRS: Marine Video Retrieval System with Explainability via Semantic Understanding [11.878077736295863]
  MarineVRS is a novel and flexible video retrieval system designed explicitly for the marine domain. MarineVRS integrates state-of-the-art methods for visual and linguistic object representation to enable efficient and accurate search and analysis of vast volumes of underwater video data. MarineVRS is a powerful tool for marine researchers and scientists to efficiently and accurately process vast amounts of data and gain deeper insights into the behavior and movements of marine species.
  arXiv  Detail & Related papers  (2023-06-07T16:46:44Z)
• Marine Video Kit: A New Marine Video Dataset for Content-based Analysis and Retrieval [10.526705651297146]
  In this paper, we focus on single-shot videos taken from moving cameras in underwater environments. The first shard of a new Marine Video Kit is presented to serve for video retrieval and other computer vision challenges.
  arXiv  Detail & Related papers  (2022-09-23T10:57:50Z)
• Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions [75.77044856100349]
  We present the Spoken Moments dataset of 500k spoken captions each attributed to a unique short video depicting a broad range of different events. We show that our AMM approach consistently improves our results and that models trained on our Spoken Moments dataset generalize better than those trained on other video-caption datasets.
  arXiv  Detail & Related papers  (2021-05-10T16:30:46Z)
• A Local-to-Global Approach to Multi-modal Movie Scene Segmentation [95.34033481442353]
  We build a large-scale video dataset MovieScenes, which contains 21K annotated scene segments from 150 movies. We propose a local-to-global scene segmentation framework, which integrates multi-modal information across three levels, i.e. clip, segment, and movie. Our experiments show that the proposed network is able to segment a movie into scenes with high accuracy, consistently outperforming previous methods.
  arXiv  Detail & Related papers  (2020-04-06T13:58:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.