Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generation

• URL: http://arxiv.org/abs/2503.24379v1
• Date: Mon, 31 Mar 2025 17:59:01 GMT
• Title: Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generation
• Authors: Shengqiong Wu, Weicai Ye, Jiahao Wang, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Shuicheng Yan, Hao Fei, Tat-Seng Chua,
• Abstract summary: We present Any2Caption, a novel framework for controllable video generation under any condition.<n>By leveraging modern multimodal large language models (MLLMs), Any2Caption interprets diverse inputs--text, images, videos, and specialized cues such as region, motion, and camera poses--into dense, structured captions.<n> Comprehensive evaluations demonstrate significant improvements of our system in controllability and video quality across various aspects of existing video generation models.
• Score: 118.5096631571738
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: To address the bottleneck of accurate user intent interpretation within the current video generation community, we present Any2Caption, a novel framework for controllable video generation under any condition. The key idea is to decouple various condition interpretation steps from the video synthesis step. By leveraging modern multimodal large language models (MLLMs), Any2Caption interprets diverse inputs--text, images, videos, and specialized cues such as region, motion, and camera poses--into dense, structured captions that offer backbone video generators with better guidance. We also introduce Any2CapIns, a large-scale dataset with 337K instances and 407K conditions for any-condition-to-caption instruction tuning. Comprehensive evaluations demonstrate significant improvements of our system in controllability and video quality across various aspects of existing video generation models. Project Page: https://sqwu.top/Any2Cap/

Related papers

• CamContextI2V: Context-aware Controllable Video Generation [12.393723748030235]
  CamContextI2V integrates multiple image conditions with 3D constraints alongside camera control to enrich both global semantics and fine-grained visual details. Our comprehensive study on the RealEstate10K dataset demonstrates improvements in visual quality and camera controllability.
  arXiv  Detail & Related papers  (2025-04-08T13:26:59Z)
• VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation [44.05151169366881]
  This paper introduces VidCapBench, a video caption evaluation scheme specifically designed for T2V generation.<n>VidCapBench associates each collected video with key information spanning video aesthetics, content, motion, and physical laws.<n>We demonstrate the superior stability and comprehensiveness of VidCapBench compared to existing video captioning evaluation approaches.
  arXiv  Detail & Related papers  (2025-02-18T11:42:17Z)
• Factorized-Dreamer: Training A High-Quality Video Generator with Limited and Low-Quality Data [14.489919164476982]
  High-quality (HQ) video synthesis is challenging because of the diverse and complex motions existed in real world. Most existing works struggle to address this problem by collecting large-scale captions, which are inaccessible to the community. We show that publicly available limited and low-quality (LQ) data are sufficient to train a HQ video generator without recaptioning or finetuning.
  arXiv  Detail & Related papers  (2024-08-19T16:08:00Z)
• When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding [118.72266141321647]
  Cross-Modality Video Coding (CMVC) is a pioneering approach to explore multimodality representation and video generative models in video coding.<n>During decoding, previously encoded components and video generation models are leveraged to create multiple encoding-decoding modes.<n>Experiments indicate that TT2V achieves effective semantic reconstruction, while IT2V exhibits competitive perceptual consistency.
  arXiv  Detail & Related papers  (2024-08-15T11:36:18Z)
• ShareGPT4Video: Improving Video Understanding and Generation with Better Captions [93.29360532845062]
  We present the ShareGPT4Video series, aiming to facilitate the video understanding of large video-language models (LVLMs) and the video generation of text-to-video models (T2VMs) via dense and precise captions. The series comprises: ShareGPT4Video, 40K GPT4V annotated dense captions of videos with various lengths and sources, developed through carefully designed data filtering and annotating strategy. We further develop ShareCaptioner-Video, a superior captioner capable of efficiently generating high-quality captions for arbitrary videos.
  arXiv  Detail & Related papers  (2024-06-06T17:58:54Z)
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning [93.6842670770983]
  Vid2Seq is a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale. We show that it is possible to leverage unlabeled narrated videos for dense video captioning, by reformulating sentence boundaries of transcribed speech as pseudo event boundaries. The resulting Vid2Seq model pretrained on the YT-Temporal-1B dataset improves the state of the art on a variety of dense video captioning benchmarks.
  arXiv  Detail & Related papers  (2023-02-27T19:53:49Z)
• Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning [36.85533835408882]
  This work presents a multimodal video generation framework that benefits from text and images provided jointly or separately. We propose a new video token trained with self-learning and an improved mask-prediction algorithm for sampling video tokens. Our framework can incorporate various visual modalities, such as segmentation masks, drawings, and partially occluded images.
  arXiv  Detail & Related papers  (2022-03-04T21:09:13Z)
• DVCFlow: Modeling Information Flow Towards Human-like Video Captioning [163.71539565491113]
  Existing methods mainly generate captions from individual video segments, lacking adaptation to the global visual context. We introduce the concept of information flow to model the progressive information changing across video sequence and captions. Our method significantly outperforms competitive baselines, and generates more human-like text according to subject and objective tests.
  arXiv  Detail & Related papers  (2021-11-19T10:46:45Z)
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training [75.55823420847759]
  We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions.
  arXiv  Detail & Related papers  (2020-05-01T03:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.