Building a Precise Video Language with Human-AI Oversight

• URL: http://arxiv.org/abs/2604.21718v2
• Date: Sun, 26 Apr 2026 23:28:29 GMT
• Title: Building a Precise Video Language with Human-AI Oversight
• Abstract summary: CHAI (Critique-based Human-AI Oversight) is a framework where trained experts critique and revise model-language pre-captions into improved post-captions.<n>Our ablations show that critique quality in precision, recall, and constructiveness, ensured by our oversight framework, directly governs downstream performance.
• Score: 63.57293532815199
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video-language models (VLMs) learn to reason about the dynamic visual world through natural language. We introduce a suite of open datasets, benchmarks, and recipes for scalable oversight that enable precise video captioning. First, we define a structured specification for describing subjects, scenes, motion, spatial, and camera dynamics, grounded by hundreds of carefully defined visual primitives developed with professional video creators such as filmmakers. Next, to curate high-quality captions, we introduce CHAI (Critique-based Human-AI Oversight), a framework where trained experts critique and revise model-generated pre-captions into improved post-captions. This division of labor improves annotation accuracy and efficiency by offloading text generation to models, allowing humans to better focus on verification. Additionally, these critiques and preferences between pre- and post-captions provide rich supervision for improving open-source models (Qwen3-VL) on caption generation, reward modeling, and critique generation through SFT, DPO, and inference-time scaling. Our ablations show that critique quality in precision, recall, and constructiveness, ensured by our oversight framework, directly governs downstream performance. With modest expert supervision, the resulting model outperforms closed-source models such as Gemini-3.1-Pro. Finally, we apply our approach to re-caption large-scale professional videos (e.g., films, commercials, games) and fine-tune video generation models such as Wan to better follow detailed prompts of up to 400 words, achieving finer control over cinematography including camera motion, angle, lens, focus, point of view, and framing. Our results show that precise specification and human-AI oversight are key to professional-level video understanding and generation. Data and code are available on our project page: https://linzhiqiu.github.io/papers/chai/

Related papers

• Video Generation Models are General-Purpose Vision Learners [62.28137589984863]
  We argue that large-scale text-to-video generation serves as a strong pre-training paradigm for computer vision.<n>We introduce GenCeption, which leverages a pre-trained video video diffusion backbone to define a feed-forward perception model.<n> Empirical results demonstrate that GenCeption achieves state-of-the-art performance across a diverse suite of tasks.
  arXiv  Detail & Related papers  (2026-07-10T01:09:06Z)
• Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence [73.39562379177796]
  LingBot-Video is a DiT-based video pretraining paradigm specifically tailored for embodied intelligence.<n>From the data perspective, we construct a data profiling engine that augments standard internet videos with extensive robot-oriented footage.<n>From the training perspective, we develop a multi-dimensional reward system to enforce the alignment regarding physical rationality and task completion.
  arXiv  Detail & Related papers  (2026-07-08T17:33:40Z)
• AVC-DPO: Aligned Video Captioning via Direct Preference Optimization [50.08618093204503]
  Video multimodal large language models (video MLLMs) have achieved substantial progress in video captioning tasks.<n>We propose Aligned Video Captioning via Direct Preference Optimization (AVC-DPO), a post-training framework designed to enhance captioning capabilities in video MLLMs through preference alignment.<n>We have achieved exceptional performance in the LOVE@PRCV'25 Workshop Track 1A: Video Detailed Captioning Challenge, achieving first place on the Video Detailed Captioning benchmark.
  arXiv  Detail & Related papers  (2025-07-02T08:51:45Z)
• ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models [87.43784424444128]
  We introduce ShotBench, a benchmark specifically designed for cinematic language understanding.<n>It features over 3.5k expert-annotated QA pairs from images and video clips, meticulously curated from over 200 acclaimed (predominantly Oscar-nominated) films.<n>Our evaluation of 24 leading Vision-Language Models on ShotBench reveals their substantial limitations, particularly struggling with fine-grained visual cues and complex spatial reasoning.
  arXiv  Detail & Related papers  (2025-06-26T15:09:21Z)
• Perception Encoder: The best visual embeddings are not at the output of the network [70.86738083862099]
  We introduce Perception (PE), a vision encoder for image and video understanding trained via simple vision-language learning.<n>We find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks.<n>Together, our PE family of models achieves best-in-class results on a wide variety of tasks.
  arXiv  Detail & Related papers  (2025-04-17T17:59:57Z)
• VideoAuteur: Towards Long Narrative Video Generation [22.915448471769384]
  We present a large-scale cooking video dataset designed to advance long-form narrative generation in the cooking domain.<n>We introduce a Long Narrative Video Director to enhance both visual and semantic coherence in generated videos.<n>Our method demonstrates substantial improvements in generating visually detailed and semantically aligneds.
  arXiv  Detail & Related papers  (2025-01-10T18:52:11Z)
• Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decoupling [79.49128866877922]
  Video-Teller is a video-language foundation model that leverages multi-modal fusion and fine-grained modality alignment. Video-Teller boosts the training efficiency by utilizing frozen pretrained vision and language modules. It capitalizes on the robust linguistic capabilities of large language models, enabling the generation of both concise and elaborate video descriptions.
  arXiv  Detail & Related papers  (2023-10-08T03:35:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.