Building Scalable Video Understanding Benchmarks through Sports

• URL: http://arxiv.org/abs/2301.06866v3
• Date: Sun, 26 Mar 2023 08:34:56 GMT
• Title: Building Scalable Video Understanding Benchmarks through Sports
• Abstract summary: We introduce an automated. Video Stream Alignment Pipeline (abbreviated ASAP) We demonstrate the generality of ASAP by aligning unlabeled videos of four different sports with corresponding dense web annotations. We then leverage ASAP scalability to create LCric, a large-scale long video understanding benchmark.
• Score: 22.851883205547644
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing benchmarks for evaluating long video understanding falls short on two critical aspects, either lacking in scale or quality of annotations. These limitations arise from the difficulty in collecting dense annotations for long videos, which often require manually labeling each frame. In this work, we introduce an automated Annotation and Video Stream Alignment Pipeline (abbreviated ASAP). We demonstrate the generality of ASAP by aligning unlabeled videos of four different sports with corresponding freely available dense web annotations (i.e. commentary). We then leverage ASAP scalability to create LCric, a large-scale long video understanding benchmark, with over 1000 hours of densely annotated long Cricket videos (with an average sample length of ~50 mins) collected at virtually zero annotation cost. We benchmark and analyze state-of-the-art video understanding models on LCric through a large set of compositional multi-choice and regression queries. We establish a human baseline that indicates significant room for new research to explore. Our human studies indicate that ASAP can align videos and annotations with high fidelity, precision, and speed. The dataset along with the code for ASAP and baselines can be accessed here: https://asap-benchmark.github.io/.

Related papers

• VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding [27.284730260765883]
  VideoOdyssey is a benchmark for ultra-long-context and omni-modal video understanding.<n>It spans 11 domains and 54 subcategories with an average video duration of 109 minutes.<n>It provides a comprehensive diagnostic tool to evaluate models across varying context lengths and cognitive loads.
  arXiv  Detail & Related papers  (2026-05-21T18:00:22Z)
• LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling [87.98096428508181]
  LongVT is an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought.<n>We exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames.<n>Our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning.
  arXiv  Detail & Related papers  (2025-11-25T19:22:48Z)
• Unleashing Hour-Scale Video Training for Long Video-Language Understanding [61.717205915329664]
  We present VideoMarathon, a large-scale hour-long video instruction-following dataset.<n>This dataset includes around 9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60 minutes per video.<n>We propose Hour-LLaVA, a powerful and efficient Video-LMM for hour-scale video-language modeling.
  arXiv  Detail & Related papers  (2025-06-05T17:59:04Z)
• TextVidBench: A Benchmark for Long Video Scene Text Understanding [60.94150574231576]
  We introduce TextVidBench, the first benchmark specifically designed for long-video text question answering (>3 minutes)<n>TextVidBench makes three key contributions: Spanning 9 categories (e.g., news, sports, gaming), with an average video length of 2306 seconds, enabling more realistic evaluation of long-video understanding.<n>We propose an efficient paradigm for improving large models through: (i) introducing the IT-Rope mechanism and temporal prompt engineering to enhance temporal perception, (ii) adopting non-uniform positional encoding to better handle long video sequences, and (iii) applying lightweight fine-tuning on
  arXiv  Detail & Related papers  (2025-06-05T12:54:56Z)
• SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models [80.3895950009792]
  Achieving fine-grained-temporal understanding in videos remains a major challenge for current Video Large Multimodels (Video LMMs)<n>We contribute in three core aspects: dataset, model, and benchmark.<n>First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically to enable joint learning of video understanding, grounding, and multi-turn video chat.<n>Second, we propose the SAMA model, which incorporates a versatile-temporal context aggregator and a Segment Model to jointly enhance fine-grained video comprehension and precise grounding capabilities.
  arXiv  Detail & Related papers  (2025-05-24T18:13:16Z)
• HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding [52.696422425058245]
  We build a large-scale hour-long long video benchmark, HLV-1K, designed to evaluate long video understanding models. HLV-1K comprises 1009 hour-long videos with 14,847 high-quality question answering (QA) and multi-choice question asnwering (MCQA) We evaluate our benchmark using existing state-of-the-art methods and demonstrate its value for testing deep long video understanding capabilities at different levels and for various tasks.
  arXiv  Detail & Related papers  (2025-01-03T05:32:37Z)
• Can LVLMs Describe Videos like Humans? A Five-in-One Video Annotations Benchmark for Better Human-Machine Comparison [15.363132825156477]
  Video description serves as a fundamental task for evaluating video comprehension, requiring a deep understanding of spatial and temporal dynamics. Current benchmarks for video comprehension have notable limitations, including short video durations, brief annotations, and reliance on a single annotator's perspective. We propose a novel benchmark, FIOVA, designed to evaluate the differences between LVLMs and human understanding more comprehensively.
  arXiv  Detail & Related papers  (2024-10-20T03:59:54Z)
• LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding [41.9477837230283]
  LongVideoBench is a question-answering benchmark that features video-language interleaved inputs up to an hour long. Our benchmark includes 3,763 varying-length web-collected videos with their subtitles across diverse themes. We formulate a novel video question-answering task termed referring reasoning.
  arXiv  Detail & Related papers  (2024-07-22T16:00:55Z)
• Goldfish: Vision-Language Understanding of Arbitrarily Long Videos [51.547065479762715]
  We present a methodology tailored for comprehending videos of arbitrary lengths. We also introduce the TVQA-long benchmark, designed to evaluate models' capabilities in understanding long videos with questions in both vision and text content. Our results indicate that our models have significant improvements in both long and short-video understanding.
  arXiv  Detail & Related papers  (2024-07-17T15:59:32Z)
• Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs [20.168429351519055]
  Video understanding is a crucial next step for multimodal large language models (LMLMs) We propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation. We conduct a comprehensive evaluation of both proprietary and open-source models, uncovering significant differences in their video understanding capabilities.
  arXiv  Detail & Related papers  (2024-06-13T17:50:05Z)
• LVBench: An Extreme Long Video Understanding Benchmark [38.839913137854104]
  We introduce LVBench, a benchmark specifically designed for long video understanding. Our dataset comprises publicly sourced videos and encompasses a diverse set of tasks aimed at long video comprehension and information extraction.
  arXiv  Detail & Related papers  (2024-06-12T09:36:52Z)
• Koala: Key frame-conditioned long video-LLM [70.52369588364992]
  We propose a lightweight and self-supervised long video-LLM (Koala) to adapt pretrained vLLMs for generalizing to longer videos. Our approach outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across all tasks. Surprisingly, we also empirically show that our approach not only helps a pretrained vLLM to understand long videos but also improves its accuracy on short-term action recognition.
  arXiv  Detail & Related papers  (2024-04-05T18:33:04Z)
• LongVLM: Efficient Long Video Understanding via Large Language Models [55.813206751150716]
  LongVLM is a simple yet powerful VideoLLM for long video understanding. We encode video representations that incorporate both local and global information. Our model produces more precise responses for long video understanding.
  arXiv  Detail & Related papers  (2024-04-04T11:33:29Z)
• LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answering [49.68215536040896]
  LvBench is a long-form video understanding benchmark for versatile multi-modal question-answering.<n>We consider videos ranging from 70 seconds to 4 hours, covering single-scene, multi-scene, and full-scene contexts.<n>Our dataset comprises 20,061 question-answer pairs sourced from 100 carefully selected movies.
  arXiv  Detail & Related papers  (2023-12-08T03:33:38Z)
• Hierarchical Video-Moment Retrieval and Step-Captioning [68.4859260853096]
  HiREST consists of 3.4K text-video pairs from an instructional video dataset. Our hierarchical benchmark consists of video retrieval, moment retrieval, and two novel moment segmentation and step captioning tasks.
  arXiv  Detail & Related papers  (2023-03-29T02:33:54Z)
• TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency [133.75876535332003]
  We focus on summarizing instructional videos, an under-explored area of video summarization. Existing video summarization datasets rely on manual frame-level annotations. We propose an instructional video summarization network that combines a context-aware temporal video encoder and a segment scoring transformer.
  arXiv  Detail & Related papers  (2022-08-14T04:07:40Z)
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling [98.41300980759577]
  A canonical approach to video-and-language learning dictates a neural model to learn from offline-extracted dense video features. We propose a generic framework ClipBERT that enables affordable end-to-end learning for video-and-language tasks. Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms existing methods.
  arXiv  Detail & Related papers  (2021-02-11T18:50:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.