SnAG: Scalable and Accurate Video Grounding

• URL: http://arxiv.org/abs/2404.02257v2
• Date: Fri, 5 Apr 2024 17:02:31 GMT
• Title: SnAG: Scalable and Accurate Video Grounding
• Abstract summary: Temporal grounding of text descriptions in videos is a central problem in vision-language learning and video understanding. We study the effect of cross-modal fusion on the scalability of video grounding models. We present SnAG, a simple baseline for scalable and accurate video grounding.
• Score: 10.578025234151596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Temporal grounding of text descriptions in videos is a central problem in vision-language learning and video understanding. Existing methods often prioritize accuracy over scalability -- they have been optimized for grounding only a few text queries within short videos, and fail to scale up to long videos with hundreds of queries. In this paper, we study the effect of cross-modal fusion on the scalability of video grounding models. Our analysis establishes late fusion as a more cost-effective fusion scheme for long-form videos with many text queries. Moreover, it leads us to a novel, video-centric sampling scheme for efficient training. Based on these findings, we present SnAG, a simple baseline for scalable and accurate video grounding. Without bells and whistles, SnAG is 43% more accurate and 1.5x faster than CONE, a state of the art for long-form video grounding on the challenging MAD dataset, while achieving highly competitive results on short videos.

Related papers

• Evidence-Backed Video Question Answering [90.93225758130426]
  Existing explainability efforts rely on textual rationales or sparse bounding boxes.<n>We propose Evidence-Backed Video Question Answering (E-VQA)<n>To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding.
  arXiv  Detail & Related papers  (2026-07-13T17:49:10Z)
• SVHighlights: Towards Extremely Long Sport Video Highlight Detection [13.468112362451093]
  We introduce SVHighlights, the first benchmark for highlight detection in extremely long sports videos.<n>The benchmark comprises 320 videos with an average duration of 2.00 hours and a total of 640.18 hours, substantially exceeding previous datasets.<n>We present TF-SELECTOR, a training-free segment-based approach that divides each video into context-aware segments.
  arXiv  Detail & Related papers  (2026-06-05T05:42:19Z)
• USV: Towards Understanding the User-generated Short-form Videos [36.991933865213355]
  This paper presents USV, the User-generated Short-form Video dataset for high-level semantic video understanding.<n>The dataset contains around 224K videos collected from platforms by label queries without extra manual verification and trimming.
  arXiv  Detail & Related papers  (2026-05-20T07:27:36Z)
• LongVideo-R1: Smart Navigation for Low-cost Long Video Understanding [106.23494088118571]
  LongVideo-R1 is a multimodal large language model (MLLM) agent for efficient video context navigation.<n>It infers the most informative video clip for subsequent processing.<n>The LongVideo-R1 agent is fine-tuned upon the Qwen-3-8B model through a two-stage paradigm.
  arXiv  Detail & Related papers  (2026-02-24T13:49:47Z)
• Think with Grounding: Curriculum Reinforced Reasoning with Video Grounding for Long Video Understanding [38.87967229483403]
  Video-TwG is a curriculum reinforced framework that employs a novel Think-with-Grounding paradigm.<n>Video-TwG can be trained end-to-end in a straightforward manner, without relying on complex auxiliary modules or heavily annotated reasoning traces.<n>Our algorithm features the fine-grained grounding reward, self-confirmed pseudo reward and accuracy-gated mechanism.
  arXiv  Detail & Related papers  (2026-02-21T03:16:23Z)
• VideoWeave: A Data-Centric Approach for Efficient Video Understanding [54.5804686337209]
  We present VideoWeave, a simple yet effective approach to improve data efficiency by constructing synthetic long-context training samples.<n>VideoWeave reorganizes available video-text pairs to expand temporal diversity within fixed compute.<n>Our results highlight that reorganizing training data, rather than altering architectures, may offer a simple and scalable path for training video-language models.
  arXiv  Detail & Related papers  (2026-01-09T20:55:26Z)
• LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling [87.98096428508181]
  LongVT is an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought.<n>We exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames.<n>Our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning.
  arXiv  Detail & Related papers  (2025-11-25T19:22:48Z)
• Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning [29.811030252357195]
  multimodal large language models (MLLMs) are crucial for downstream tasks like video question answering and temporal grounding.<n>We propose Video Intelligence via Tool-Augmented Learning (VITAL), a novel end-to-end agentic video reasoning framework.
  arXiv  Detail & Related papers  (2025-08-06T13:03:21Z)
• ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts [56.75723197779384]
  ARC-Hunyuan-Video is a multimodal model that processes visual, audio, and textual signals end-to-end for structured comprehension.<n>Our model is capable of multi-granularity timestamped video captioning and summarization, open-ended video question answering, temporal video grounding, and video reasoning.
  arXiv  Detail & Related papers  (2025-07-28T15:52:36Z)
• Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models [53.235170710385006]
  We introduce Grounded-VideoLLM, a novel Video-LLM adept at perceiving and reasoning over specific video moments in a fine-grained manner. We sharpen our model by incorporating (1) an additional temporal stream to encode the relationships between frames and (2) discrete temporal tokens enriched with specific time knowledge. In experiments, Grounded-VideoLLM excels in fine-grained grounding tasks such as temporal sentence grounding, dense video captioning, and grounded VideoQA.
  arXiv  Detail & Related papers  (2024-10-04T10:04:37Z)
• SynopGround: A Large-Scale Dataset for Multi-Paragraph Video Grounding from TV Dramas and Synopses [58.488812405557]
  Video grounding aims to localize specific natural language queries in an untrimmed video. We present a large-scale video grounding dataset named SynopGround. We introduce a more complex setting of video grounding dubbed Multi-Paragraph Video Grounding (MPVG)
  arXiv  Detail & Related papers  (2024-08-03T05:35:13Z)
• DrVideo: Document Retrieval Based Long Video Understanding [44.34473173458403]
  DrVideo is a document-retrieval-based system designed for long video understanding. It first transforms a long video into a coarse text-based long document to retrieve key frames and then updates the documents with the augmented key frame information. It then employs an agent-based iterative loop to continuously search for missing information and augment the document until sufficient question-related information is gathered.
  arXiv  Detail & Related papers  (2024-06-18T17:59:03Z)
• AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction [88.70116693750452]
  Text-guided video prediction (TVP) involves predicting the motion of future frames from the initial frame according to an instruction. Previous TVP methods make significant breakthroughs by adapting Stable Diffusion for this task. We introduce the Multi-Modal Large Language Model (MLLM) to predict future video states based on initial frames and text instructions.
  arXiv  Detail & Related papers  (2024-06-10T17:02:08Z)
• Koala: Key frame-conditioned long video-LLM [70.52369588364992]
  We propose a lightweight and self-supervised long video-LLM (Koala) to adapt pretrained vLLMs for generalizing to longer videos. Our approach outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across all tasks. Surprisingly, we also empirically show that our approach not only helps a pretrained vLLM to understand long videos but also improves its accuracy on short-term action recognition.
  arXiv  Detail & Related papers  (2024-04-05T18:33:04Z)
• Scanning Only Once: An End-to-end Framework for Fast Temporal Grounding in Long Videos [60.86880787242561]
  Video temporal grounding aims to pinpoint a video segment that matches the query description. We propose an end-to-end framework for fast temporal grounding, which is able to model an hours-long video with textbfone-time network execution. Our method significantly outperforms state-of-the-arts, and achieves textbf14.6$times$ / textbf102.8$times$ higher efficiency respectively.
  arXiv  Detail & Related papers  (2023-03-15T03:54:43Z)
• CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding [70.7882058229772]
  This paper tackles an emerging and challenging problem of long video temporal grounding(VTG) Compared with short videos, long videos are also highly demanded but less explored. We propose CONE, an efficient COarse-to-fiNE alignment framework.
  arXiv  Detail & Related papers  (2022-09-22T10:58:42Z)
• Text-Driven Video Acceleration: A Weakly-Supervised Reinforcement Learning Method [6.172652648945223]
  This paper presents a novel weakly-supervised methodology to accelerate instructional videos using text. A novel joint reward function guides our agent to select which frames to remove and reduce the input video to a target length. We also propose the Extended Visually-guided Document Attention Network (VDAN+), which can generate a highly discriminative embedding space.
  arXiv  Detail & Related papers  (2022-03-29T17:43:01Z)
• Few-Shot Video Object Detection [70.43402912344327]
  We introduce Few-Shot Video Object Detection (FSVOD) with three important contributions. FSVOD-500 comprises of 500 classes with class-balanced videos in each category for few-shot learning. Our TPN and TMN+ are jointly and end-to-end trained.
  arXiv  Detail & Related papers  (2021-04-30T07:38:04Z)
• Straight to the Point: Fast-forwarding Videos via Reinforcement Learning Using Textual Data [1.004766879203303]
  We present a novel methodology based on a reinforcement learning formulation to accelerate instructional videos. Our approach can adaptively select frames that are not relevant to convey the information without creating gaps in the final video. We propose a novel network, called Visually-guided Document Attention Network (VDAN), able to generate a highly discriminative embedding space.
  arXiv  Detail & Related papers  (2020-03-31T14:07:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.