HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation

• URL: http://arxiv.org/abs/2503.23715v1
• Date: Mon, 31 Mar 2025 04:30:34 GMT
• Title: HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation
• Authors: Kun Liu, Qi Liu, Xinchen Liu, Jie Li, Yongdong Zhang, Jiebo Luo, Xiaodong He, Wu Liu,
• Abstract summary: We introduce HOIGen-1M, the first largescale dataset for HOI Generation, consisting of over one million high-quality videos.<n>To guarantee the high quality of videos, we first design an efficient framework to automatically curate HOI videos using the powerful multimodal large language models (MLLMs)<n>To obtain accurate textual captions for HOI videos, we design a novel video description method based on a Mixture-of-Multimodal-Experts (MoME) strategy.
• Score: 99.6653979969241
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-video (T2V) generation has made tremendous progress in generating complicated scenes based on texts. However, human-object interaction (HOI) often cannot be precisely generated by current T2V models due to the lack of large-scale videos with accurate captions for HOI. To address this issue, we introduce HOIGen-1M, the first largescale dataset for HOI Generation, consisting of over one million high-quality videos collected from diverse sources. In particular, to guarantee the high quality of videos, we first design an efficient framework to automatically curate HOI videos using the powerful multimodal large language models (MLLMs), and then the videos are further cleaned by human annotators. Moreover, to obtain accurate textual captions for HOI videos, we design a novel video description method based on a Mixture-of-Multimodal-Experts (MoME) strategy that not only generates expressive captions but also eliminates the hallucination by individual MLLM. Furthermore, due to the lack of an evaluation framework for generated HOI videos, we propose two new metrics to assess the quality of generated videos in a coarse-to-fine manner. Extensive experiments reveal that current T2V models struggle to generate high-quality HOI videos and confirm that our HOIGen-1M dataset is instrumental for improving HOI video generation. Project webpage is available at https://liuqi-creat.github.io/HOIGen.github.io.

Related papers

• VideoAuteur: Towards Long Narrative Video Generation [22.915448471769384]
  We present a large-scale cooking video dataset designed to advance long-form narrative generation in the cooking domain. We introduce a Long Narrative Video Director to enhance both visual and semantic coherence in generated videos. Our method demonstrates substantial improvements in generating visually detailed and semantically aligneds.
  arXiv  Detail & Related papers  (2025-01-10T18:52:11Z)
• xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations [120.52120919834988]
  xGen-SynVideo-1 is a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions. VidVAE compresses video data both spatially and temporally, significantly reducing the length of visual tokens. DiT model incorporates spatial and temporal self-attention layers, enabling robust generalization across different timeframes and aspect ratios.
  arXiv  Detail & Related papers  (2024-08-22T17:55:22Z)
• OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation [33.62365864717086]
  We introduce OpenVid-1M, a precise high-quality dataset with expressive captions.<n>We also curate 433K 1080p videos from OpenVid-1M to create OpenVidHD-0.4M, advancing high-definition video generation.
  arXiv  Detail & Related papers  (2024-07-02T15:40:29Z)
• Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment [54.00254267259069]
  We establish the largest-scale Text-to-Video Quality Assessment DataBase (T2VQA-DB) to date. The dataset is composed of 10,000 videos generated by 9 different T2V models. We propose a novel transformer-based model for subjective-aligned Text-to-Video Quality Assessment (T2VQA)
  arXiv  Detail & Related papers  (2024-03-18T16:52:49Z)
• Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization [52.63845811751936]
  Video pre-training is challenging due to the modeling of its dynamics video. In this paper, we address such limitations in video pre-training with an efficient video decomposition. Our framework is both capable of comprehending and generating image and video content, as demonstrated by its performance across 13 multimodal benchmarks.
  arXiv  Detail & Related papers  (2024-02-05T16:30:49Z)
• Distilling Vision-Language Models on Millions of Videos [62.92789440875999]
  We fine-tune a video-language model from a strong image-language baseline with synthesized instructional data. The resulting video model by video-instruction-tuning (VIIT) is then used to auto-label millions of videos to generate high-quality captions. As a side product, we generate the largest video caption dataset to date.
  arXiv  Detail & Related papers  (2024-01-11T18:59:53Z)
• Make-A-Video: Text-to-Video Generation without Text-Video Data [69.20996352229422]
  Make-A-Video is an approach for translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V) We design a simple yet effective way to build on T2I models with novel and effective spatial-temporal modules. In all aspects, spatial and temporal resolution, faithfulness to text, and quality, Make-A-Video sets the new state-of-the-art in text-to-video generation.
  arXiv  Detail & Related papers  (2022-09-29T13:59:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.