OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding

• URL: http://arxiv.org/abs/2604.25276v1
• Date: Tue, 28 Apr 2026 06:34:19 GMT
• Title: OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding
• Abstract summary: Video Temporal Grounding (VTG) struggles in open-world settings due to limited dataset scale and semantic diversity.<n>We introduce OmniVTG, a new large-scale dataset for open-world VTG.<n>We train the MLLM to first predict, then use its understanding capabilities to reflect on and refine its own predictions.
• Score: 55.29748680163419
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Temporal Grounding (VTG), the task of localizing video segments from text queries, struggles in open-world settings due to limited dataset scale and semantic diversity, causing performance gaps between common and rare concepts. To overcome these limitations, we introduce OmniVTG, a new large-scale dataset for open-world VTG, coupled with a Self-Correction Chain-of-Thought (CoT) training paradigm designed to enhance the grounding capabilities of Multimodal Large Language Models (MLLMs). Our OmniVTG is constructed via a novel Semantic Coverage Iterative Expansion pipeline, which first identifies gaps in the vocabulary of existing datasets and collects videos that are highly likely to contain these target concepts. For high-quality annotation, we leverage the insight that modern MLLMs excel at dense captioning more than direct grounding and design a caption-centric data engine to prompt MLLMs to generate dense, timestamped descriptions. Beyond the dataset, we observe that simple supervised finetuning (SFT) is insufficient, as a performance gap between rare and common concepts still persists. We find that MLLMs' video understanding ability significantly surpasses their direct grounding ability. Based on this, we propose a Self-Correction Chain-of-Thought (CoT) training paradigm. We train the MLLM to first predict, then use its understanding capabilities to reflect on and refine its own predictions. This capability is instilled via a three-stage pipeline of SFT, CoT finetuning, and reinforcement learning. Extensive experiments show our approach not only excels at open-world grounding in our OmniVTG dataset but also achieves state-of-the-art zero-shot performance on four existing VTG benchmarks. Code is available at https://github.com/oceanflowlab/OmniVTG.

Related papers

• UniversalVTG: A Universal and Lightweight Foundation Model for Video Temporal Grounding [46.36397337493086]
  Video temporal grounding (VTG) is typically tackled with dataset-specific models that transfer poorly across domains and query styles.<n>We present UniversalVTG, a single VTG model trained with large-scale cross-dataset pretraining.
  arXiv  Detail & Related papers  (2026-04-09T17:57:09Z)
• GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding [8.196520511153368]
  Video temporal grounding (VTG) is a critical task in video and understanding a key capability for extending large language models (Vid-LLMs) to broader applications.<n>Existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues.<n>We propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments.
  arXiv  Detail & Related papers  (2026-04-02T14:19:58Z)
• TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigation [70.23578202012048]
  Vision-Language Navigation (VLN) presents a unique challenge for Large Vision-Language Models (VLMs) due to their inherent architectural mismatch.<n>We propose TagaVLM (Topology-Aware Global Action reasoning), an end-to-end framework that explicitly injects topological structures into the VLM backbone.<n>To enhance topological node information, an Interleaved Navigation Prompt strengthens node-level visual-text alignment.<n>With the embedded topological graph, the model is capable of global action reasoning, allowing for robust path correction.
  arXiv  Detail & Related papers  (2026-03-03T13:28:07Z)
• TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs [81.78017865436816]
  We present TimeLens, a systematic investigation into building MLLMs with strong video temporal grounding ability.<n>We first expose critical quality issues in existing VTG benchmarks and introduce TimeLens-Bench, comprising meticulously re-annotated versions of three popular benchmarks.<n>We also address noisy training data through an automated re-annotation pipeline, yielding TimeLens-100K, a large-scale, high-quality training dataset.
  arXiv  Detail & Related papers  (2025-12-16T18:59:58Z)
• ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding [13.295759874474767]
  We introduce Vision-Aware Speculative Decoding (ViSpec), a novel framework tailored for vision-language models (VLMs)<n>ViSpec employs a lightweight vision adaptor module to compress image tokens into a compact representation.<n>Our training strategy mitigates the risk of the draft model exploiting direct access to the target model's hidden states.
  arXiv  Detail & Related papers  (2025-09-17T11:28:58Z)
• A Survey on Video Temporal Grounding with Multimodal Large Language Model [107.24431595873808]
  Recent advancement in temporal grounding (VTG) has significantly enhanced fine-grained video understanding.<n>With superior multimodal comprehension and reasoning abilities, VTG approaches based on MLLMs (VTG-MLLMs) are gradually surpassing traditional fine-tuned methods.<n>Despite extensive surveys on general video-language understanding, comprehensive reviews specifically addressing VTG-MLLMs remain scarce.
  arXiv  Detail & Related papers  (2025-08-07T08:52:11Z)
• TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning [54.033346088090674]
  We introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability.<n>To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT.<n>This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process.
  arXiv  Detail & Related papers  (2024-10-14T13:35:47Z)
• Towards Open-World Grasping with Large Vision-Language Models [5.317624228510749]
  An open-world grasping system should be able to combine high-level contextual with low-level physical-geometric reasoning. We propose OWG, an open-world grasping pipeline that combines vision-language models with segmentation and grasp synthesis models. We conduct evaluation in cluttered indoor scene datasets to showcase OWG's robustness in grounding from open-ended language.
  arXiv  Detail & Related papers  (2024-06-26T19:42:08Z)
• Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding [108.79026216923984]
  Video grounding aims to localize a-temporal section in a video corresponding to an input text query. This paper addresses a critical limitation in current video grounding methodologies by introducing an Open-Vocabulary Spatio-Temporal Video Grounding task.
  arXiv  Detail & Related papers  (2023-12-31T13:53:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.