T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation

• URL: http://arxiv.org/abs/2407.14505v1
• Date: Fri, 19 Jul 2024 17:58:36 GMT
• Title: T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation
• Authors: Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, Xihui Liu,
• Abstract summary: We conduct the first systematic study on compositional text-to-video generation. We propose T2V-CompBench, the first benchmark tailored for compositional text-to-video generation.
• Score: 55.57459883629706
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-video (T2V) generation models have advanced significantly, yet their ability to compose different objects, attributes, actions, and motions into a video remains unexplored. Previous text-to-video benchmarks also neglect this important ability for evaluation. In this work, we conduct the first systematic study on compositional text-to-video generation. We propose T2V-CompBench, the first benchmark tailored for compositional text-to-video generation. T2V-CompBench encompasses diverse aspects of compositionality, including consistent attribute binding, dynamic attribute binding, spatial relationships, motion binding, action binding, object interactions, and generative numeracy. We further carefully design evaluation metrics of MLLM-based metrics, detection-based metrics, and tracking-based metrics, which can better reflect the compositional text-to-video generation quality of seven proposed categories with 700 text prompts. The effectiveness of the proposed metrics is verified by correlation with human evaluations. We also benchmark various text-to-video generative models and conduct in-depth analysis across different models and different compositional categories. We find that compositional text-to-video generation is highly challenging for current models, and we hope that our attempt will shed light on future research in this direction.

Related papers

• NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionality [52.08735848128973]
  We study the capability of Video-Language (VidL) models in understanding compositions between objects, attributes, actions and their relations. We propose a training method called NAVERO which utilizes video-text data augmented with negative texts to enhance composition understanding.
  arXiv  Detail & Related papers  (2024-08-18T15:27:06Z)
• Evaluation of Text-to-Video Generation Models: A Dynamics Perspective [94.2662603491163]
  Existing evaluation protocols primarily focus on temporal consistency and content continuity. We propose an effective evaluation protocol, termed DEVIL, which centers on the dynamics dimension to evaluate T2V models.
  arXiv  Detail & Related papers  (2024-07-01T08:51:22Z)
• TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation [97.96178992465511]
  We argue that generated videos should incorporate the emergence of new concepts and their relation transitions like in real-world videos as time progresses. To assess the Temporal Compositionality of video generation models, we propose TC-Bench, a benchmark of meticulously crafted text prompts, corresponding ground truth videos, and robust evaluation metrics.
  arXiv  Detail & Related papers  (2024-06-12T21:41:32Z)
• Towards A Better Metric for Text-to-Video Generation [102.16250512265995]
  Generative models have demonstrated remarkable capability in synthesizing high-quality text, images, and videos. We introduce a novel evaluation pipeline, the Text-to-Video Score (T2VScore) This metric integrates two pivotal criteria: (1) Text-Video Alignment, which scrutinizes the fidelity of the video in representing the given text description, and (2) Video Quality, which evaluates the video's overall production caliber with a mixture of experts.
  arXiv  Detail & Related papers  (2024-01-15T15:42:39Z)
• T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation [62.71574695256264]
  T2I-CompBench is a comprehensive benchmark for open-world compositional text-to-image generation. We propose several evaluation metrics specifically designed to evaluate compositional text-to-image generation. We introduce a new approach, Generative mOdel fine-tuning with Reward-driven Sample selection (GORS) to boost the compositional text-to-image generation abilities.
  arXiv  Detail & Related papers  (2023-07-12T17:59:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.