CineCap: Structured Reasoning with Spatio-Temporal Anchors for Cinematographic Video Captioning
- URL: http://arxiv.org/abs/2606.24636v1
- Date: Tue, 23 Jun 2026 14:29:40 GMT
- Title: CineCap: Structured Reasoning with Spatio-Temporal Anchors for Cinematographic Video Captioning
- Abstract summary: Cinematographic captioning aims to describe how a video is filmed using film-language concepts such as camera, shot size, depth of field, composition, and shooting angle.<n>This capability is important for fine-grained video understanding and controllable movie-quality video generation, yet remains underexplored in existing multimodal large language models.<n>We propose CineCap, a framework that combines structured reasoning with atomic-temporal anchors and reinforcement learning with comprehensiveness, accuracy, and coverage rewards.
- Score: 17.457944487441697
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Cinematographic captioning aims to describe how a video is filmed using professional film-language concepts such as camera movement, shot size, depth of field, composition, and shooting angle. This capability is important for fine-grained video understanding and controllable movie-quality video generation, yet remains underexplored in existing multimodal large language models. Unlike question-answering-based evaluation of cinematic understanding, cinematographic captioning requires a unified open-form description over multiple cinematographic dimensions. This task is challenging for two main reasons: the model must infer professional cinematographic concepts from subtle visual evidence, and it must generate captions that are both comprehensive and accurate. Accordingly, we propose CineCap, a framework that combines structured reasoning with spatio-temporal anchors and reinforcement learning with comprehensiveness, accuracy, and gated coverage rewards. The former grounds professional cinematographic descriptions in explicit visual evidence and organizes them into compact atomic reasoning for supervised fine-tuning, while the latter improves the balance between descriptive completeness and factual correctness. In addition, we construct CineCap Bench, a benchmark of 472 manually annotated video-caption pairs for systematic evaluation. Extensive experiments show that CineCap consistently outperforms strong proprietary and open-source baselines, establishing a new state of the art for cinematographic captioning. The code, model checkpoint, and benchmark are publicly available in https://github.com/Hectormxy/CineCap.git.
Related papers
- Building a Precise Video Language with Human-AI Oversight [63.57293532815199]
CHAI (Critique-based Human-AI Oversight) is a framework where trained experts critique and revise model-language pre-captions into improved post-captions.<n>Our ablations show that critique quality in precision, recall, and constructiveness, ensured by our oversight framework, directly governs downstream performance.
arXiv Detail & Related papers (2026-04-22T09:01:04Z) - CutClaw: Agentic Hours-Long Video Editing via Music Synchronization [96.62825277039117]
CutClaw is an autonomous multi-agent framework designed to edit hours-long raw footage into meaningful short videos.<n>It produces videos with synchronized music, followed by instructions, and a visually appealing appearance.<n>We conduct detailed experiments to demonstrate that CutClaw significantly outperforms state-of-the-art baselines in generating high-quality, rhythm-aligned videos.
arXiv Detail & Related papers (2026-03-31T12:25:53Z) - Captain Cinema: Towards Short Movie Generation [66.22442526026215]
We present Captain Cinema, a generation framework for short movie generation.<n>Our approach generates a sequence of synthesiss that outline the entire narrative.<n>Our model is trained on a specially curated dataset consisting of interleaved data pairs.
arXiv Detail & Related papers (2025-07-24T17:59:56Z) - ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models [87.43784424444128]
We introduce ShotBench, a benchmark specifically designed for cinematic language understanding.<n>It features over 3.5k expert-annotated QA pairs from images and video clips, meticulously curated from over 200 acclaimed (predominantly Oscar-nominated) films.<n>Our evaluation of 24 leading Vision-Language Models on ShotBench reveals their substantial limitations, particularly struggling with fine-grained visual cues and complex spatial reasoning.
arXiv Detail & Related papers (2025-06-26T15:09:21Z) - CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation [22.88243961225531]
CineTechBench is a benchmark founded on precise, manual annotation by seasoned cinematography experts.<n>Our benchmark covers seven essential aspects-shot scale, shot angle, composition, camera movement, lighting, color, and focal length.<n>For the generation task, we assess advanced video generation models on their capacity to reconstruct cinema-quality camera movements.
arXiv Detail & Related papers (2025-05-21T06:02:39Z) - Progress-Aware Video Frame Captioning [55.23366888264651]
We propose ProgressCaptioner, a captioning model designed to capture the fine-grained temporal dynamics within an action sequence.<n>We develop the FrameCap dataset to support training and the FrameCapEval benchmark to assess caption quality.<n>Results demonstrate that ProgressCaptioner significantly surpasses leading captioning models.
arXiv Detail & Related papers (2024-12-03T01:21:28Z) - AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark [89.73538448786405]
We propose AuroraCap, a video captioner based on a large multimodal model.<n>We implement the token merging strategy, reducing the number of input visual tokens.<n>AuroraCap shows superior performance on various video and image captioning benchmarks.
arXiv Detail & Related papers (2024-10-04T00:13:54Z) - MoviePuzzle: Visual Narrative Reasoning through Multimodal Order Learning [47.59806301773499]
MoviePuzzle is a novel challenge that targets visual narrative reasoning and holistic movie understanding.<n>To tackle this quandary, we put forth MoviePuzzle task that amplifies the temporal feature learning and structure learning of video models.<n>Our approach outperforms existing state-of-the-art methods on the MoviePuzzle benchmark.
arXiv Detail & Related papers (2023-06-04T03:51:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.