SF20K Competition 2025: Summary and findings

• URL: http://arxiv.org/abs/2605.01496v1
• Date: Sat, 02 May 2026 15:35:54 GMT
• Title: SF20K Competition 2025: Summary and findings
• Abstract summary: Short-Films 20K (SF20K) Competition is designed to advance story-level video understanding beyond short-clip action recognition.<n>Models must rely on multimodal understanding rather than memorization of popular movies.<n>Winning team achieved 65.7% accuracy on the Main Track and 48.7% on the Special Track, against a human performance ceiling of 91.7%.
• Score: 34.86183179717155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This report presents the results and findings of the first edition of the Short-Films 20K (SF20K) Competition, held in conjunction with the SLoMO Workshop at ICCV 2025. The competition is designed to advance story-level video understanding beyond short-clip action recognition, introducing an open-ended video question-answering task built on a corpus of amateur short films. This setup ensures that models must rely on multimodal understanding rather than memorization of popular movies. Evaluation is conducted using the SF20K-Test benchmark (95 movies, 979 question-answer pairs) and scored via LLM-QA-Eval, an automated judge based on GPT-4.1-nano. The competition attracted 22 teams and 286 submissions across two tracks: a Main Track with unrestricted model size and a Special Track limited to models under 8 billion parameters. The winning team achieved 65.7% accuracy on the Main Track and 48.7% on the Special Track, against a human performance ceiling of 91.7%. Our analysis reveals several key findings: narrative-aware, shot-level processing consistently outperforms uniform frame sampling; well-designed multi-stage pipelines using smaller models can match or exceed end-to-end inference with models over 30x larger; and subtitle quality is a dominant factor in performance. These results highlight that the primary bottleneck in long-form video QA lies in information selection and reasoning structure rather than raw model capacity, and that a substantial gap remains between current methods and human-level narrative comprehension.

Related papers

• Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysis [7.392659193819963]
  Traffic safety analysis requires complex video understanding to capture behavioral patterns and generate descriptions for accident prevention.<n>In this work, we present a unique dual-model framework that strategically utilizes the complementary strengths of VideoLLaMA and Qwen2.5-VL through task-specific optimization.
  arXiv  Detail & Related papers  (2025-10-13T20:18:23Z)
• Beyond Isolated Facts: Synthesizing Narrative and Grounded Supervision for VideoQA [37.679936989592996]
  We introduce a framework to synthesize richer supervisory signals.<n>We propose two complementary strategies: Question-Based Paraphrasing (QBP) and Question-Based Captioning (QBC)
  arXiv  Detail & Related papers  (2025-09-29T08:28:44Z)
• FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting [62.25888935329454]
  This paper introduces the concept of thinking with long videos and proposes a novel framework FrameThinker.<n>We show that FrameThinker achieves a significant average improvement of +10.4% over baselines while drastically reducing the number of processed frames.<n>Most notably, our 7B model, FrameThinker establishes a new state-of-the-art on LongVideo-Reason, achieving 76.1% accuracy using an average of only 20.6 frames.
  arXiv  Detail & Related papers  (2025-09-29T05:36:58Z)
• VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos [41.9704057910616]
  We present VRBench, the first long narrative video benchmark crafted for evaluating large models' multi-step reasoning capabilities.<n>It comprises 960 long videos (with an average duration of 1.6 hours), along with 8,243 human-labeled multi-step question-answering pairs and 25,106 reasoning steps with timestamps.
  arXiv  Detail & Related papers  (2025-06-12T16:17:17Z)
• NTIRE 2025 Challenge on Short-form UGC Video Quality Assessment and Enhancement: Methods and Results [179.05961380270648]
  Review of the NTIRE 2025 Challenge on Short-form Video Quality Assessment and Enhancement.<n>Challenge comprises two tracks: (i) Efficient Video Quality Assessment (KVQ), and (ii) Diffusion-based Image Super-Resolution (KwaiSR)
  arXiv  Detail & Related papers  (2025-04-17T17:45:34Z)
• Law of the Weakest Link: Cross Capabilities of Large Language Models [102.91861246827797]
  We show that Large Language Models (LLMs) exhibit the "Law of the Weakest Link," where cross-capability performance is significantly constrained by the weakest component. These results highlight the under-performance of LLMs in cross-capability tasks.
  arXiv  Detail & Related papers  (2024-09-30T05:12:01Z)
• AIM 2024 Challenge on Video Saliency Prediction: Methods and Results [105.09572982350532]
  This paper reviews the Challenge on Video Saliency Prediction at AIM 2024. The goal of the participants was to develop a method for predicting accurate saliency maps for the provided set of video sequences.
  arXiv  Detail & Related papers  (2024-09-23T08:59:22Z)
• AIM 2024 Challenge on Compressed Video Quality Assessment: Methods and Results [120.95863275142727]
  This paper presents the results of the Compressed Video Quality Assessment challenge, held in conjunction with the Advances in Image Manipulation (AIM) workshop at ECCV 2024. The challenge aimed to evaluate the performance of VQA methods on a diverse dataset of 459 videos encoded with 14 codecs of various compression standards.
  arXiv  Detail & Related papers  (2024-08-21T20:32:45Z)
• Skating-Mixer: Multimodal MLP for Scoring Figure Skating [31.346611498891964]
  We introduce a multimodal architecture, named Skating-Mixer. It effectively learns long-term representations through our designed memory recurrent unit (MRU) Experiments show the proposed method outperforms SOTAs over all major metrics on the public Fis-V and our FS1000 dataset.
  arXiv  Detail & Related papers  (2022-03-08T10:36:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.