RefineShot: Rethinking Cinematography Understanding with Foundational Skill Evaluation

• URL: http://arxiv.org/abs/2510.02423v1
• Date: Thu, 02 Oct 2025 16:46:32 GMT
• Title: RefineShot: Rethinking Cinematography Understanding with Foundational Skill Evaluation
• Abstract summary: Cinematography understanding enhances multimodal understanding in real-world applications and underpins coherent content creation in film and media.<n>ShotBench is the most comprehensive benchmark for this task, with ShotVL achieving state-of-the-art results on it.<n>However, our analysis reveals that ambiguous option design in ShotBench and ShotVL's shortcomings in reasoning consistency and instruction adherence undermine evaluation reliability.<n>To overcome these issues, we systematically refine ShotBench through consistent option restructuring, conduct the first critical analysis of ShotVL's reasoning behavior, and introduce an extended evaluation protocol.
• Score: 53.48857764937677
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cinematography understanding refers to the ability to recognize not only the visual content of a scene but also the cinematic techniques that shape narrative meaning. This capability is attracting increasing attention, as it enhances multimodal understanding in real-world applications and underpins coherent content creation in film and media. As the most comprehensive benchmark for this task, ShotBench spans a wide range of cinematic concepts and VQA-style evaluations, with ShotVL achieving state-of-the-art results on it. However, our analysis reveals that ambiguous option design in ShotBench and ShotVL's shortcomings in reasoning consistency and instruction adherence undermine evaluation reliability, limiting fair comparison and hindering future progress. To overcome these issues, we systematically refine ShotBench through consistent option restructuring, conduct the first critical analysis of ShotVL's reasoning behavior, and introduce an extended evaluation protocol that jointly assesses task accuracy and core model competencies. These efforts lead to RefineShot, a refined and expanded benchmark that enables more reliable assessment and fosters future advances in cinematography understanding.

Related papers

• EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation [59.569634087474185]
  We introduce EvalVerse, a comprehensive, pipeline-aware, and expert-calibrated evaluation framework.<n>First, we organize domain knowledge into an evaluation taxonomy aligned with the professional filmmaking workflow.<n>Second, we distill human expert judgments into a curated dataset with large-scale human annotations.<n>Third, we inject this knowledge into Vision-Language Models (VLMs) through an expert-calibrated fine-tuning strategy.
  arXiv  Detail & Related papers  (2026-05-22T06:22:21Z)
• Q-DeepSight: Incentivizing Thinking with Images for Image Quality Assessment and Refinement [58.15004031934379]
  We propose Q-DeepSight, a think-with-image framework that emulates this human-like process.<n>Q-DeepSight achieves state-of-the-art performance across diverse benchmarks, including natural, restored, and AI-generated content.<n>We demonstrate its practical value with Perceptual-in-Generation (PiG), a training-free framework where Q-DeepSight's diagnoses guide iterative image enhancement.
  arXiv  Detail & Related papers  (2026-04-18T06:10:57Z)
• Video-Oasis: Rethinking Evaluation of Video Understanding [20.076100437038313]
  Video understanding is difficult to attribute whether performance gains stem from visual perception, linguistic reasoning or knowledge priors.<n>We provide Video-Oasis, a diagnostic suite designed to evaluate existing evaluations and distill-temporal challenges for video understanding.<n>Our analysis reveals two critical findings: (1) 54% of existing benchmark samples are solvable without visual input or temporal context, and (2) on the remaining samples, state-of-the-art models exhibit performance barely exceeding random guessing.
  arXiv  Detail & Related papers  (2026-03-31T11:37:42Z)
• VIPER: Process-aware Evaluation for Generative Video Reasoning [64.86465792516658]
  We introduce VIPER, a comprehensive benchmark spanning 16 tasks across temporal, structural, symbolic, spatial, physics, and planning reasoning.<n>Our experiments reveal that state-of-the-art video models achieve only about 20% POC@1.0 and exhibit a significant outcome-hacking.
  arXiv  Detail & Related papers  (2025-12-31T16:31:59Z)
• Q-Save: Towards Scoring and Attribution for Generated Video Evaluation [65.83319736145869]
  We present Q-Save, a new benchmark dataset and model for holistic evaluation of AI-generated video (AIGV) quality.<n>The dataset contains near 10000 videos, each annotated with a scalar mean opinion score (MOS) and fine-grained attribution labels.<n>We propose a unified evaluation model that jointly performs quality scoring and attribution-based explanation.
  arXiv  Detail & Related papers  (2025-11-24T07:00:21Z)
• VQAThinker: Exploring Generalizable and Explainable Video Quality Assessment via Reinforcement Learning [50.34205095371895]
  Video quality assessment aims to objectively quantify perceptual quality degradation.<n>Existing VQA models suffer from two critical limitations.<n>We propose textbfVQAThinker, a reasoning-based VQA framework.
  arXiv  Detail & Related papers  (2025-08-08T06:16:23Z)
• ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models [87.43784424444128]
  We introduce ShotBench, a benchmark specifically designed for cinematic language understanding.<n>It features over 3.5k expert-annotated QA pairs from images and video clips, meticulously curated from over 200 acclaimed (predominantly Oscar-nominated) films.<n>Our evaluation of 24 leading Vision-Language Models on ShotBench reveals their substantial limitations, particularly struggling with fine-grained visual cues and complex spatial reasoning.
  arXiv  Detail & Related papers  (2025-06-26T15:09:21Z)
• VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning [22.357792064450955]
  Chain-of-Thought (CoT) reasoning has significantly enhanced the capabilities of large language models (LLMs) and large vision-language models (LVLMs)<n>We introduce VCR-Bench, a novel benchmark designed to comprehensively evaluate LVLMs' Video Chain-of-Thought Reasoning capabilities.<n> VCR-Bench comprises 859 videos spanning a variety of video content and durations, along with 1,034 high-quality question-answer pairs.
  arXiv  Detail & Related papers  (2025-04-10T17:59:03Z)
• Are Large Vision Language Models Good Game Players? [25.49713745405194]
  Large Vision Language Models (LVLMs) have demonstrated remarkable abilities in understanding and reasoning about both visual and textual information.<n>Existing evaluation methods for LVLMs, primarily based on benchmarks like Visual Question Answering, often fail to capture the full scope of LVLMs' capabilities.<n>We propose method, a game-based evaluation framework designed to provide a comprehensive assessment of LVLMs' cognitive and reasoning skills in structured environments.
  arXiv  Detail & Related papers  (2025-03-04T07:29:03Z)
• STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training [87.58996020705258]
  Video Large Language Models (Video-LLMs) have recently shown strong derivation in basic video understanding tasks.<n>Video-LLMs struggle with compositional reasoning that requires multi-step explicit-temporal inference across object relations, interactions and events.<n>We propose STEP, a novel graph-guided self-training method that enables VideoLLMs to generate reasoning-rich finetuning data from any raw videos to improve itself.
  arXiv  Detail & Related papers  (2024-11-29T11:54:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.