FaceVid-1K: A Large-Scale High-Quality Multiracial Human Face Video Dataset

• URL: http://arxiv.org/abs/2410.07151v1
• Date: Mon, 23 Sep 2024 07:27:02 GMT
• Title: FaceVid-1K: A Large-Scale High-Quality Multiracial Human Face Video Dataset
• Authors: Donglin Di, He Feng, Wenzhang Sun, Yongjia Ma, Hao Li, Wei Chen, Xiaofei Gou, Tonghua Su, Xun Yang,
• Abstract summary: We create a high-quality multiracial face collection named textbfFaceVid-1K. We conduct experiments with several well-established video generation models, including text-to-video, image-to-video, and unconditional video generation. We obtain the corresponding performance benchmarks and compared them with those trained on public datasets to demonstrate the superiority of our dataset.
• Score: 15.917564646478628
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating talking face videos from various conditions has recently become a highly popular research area within generative tasks. However, building a high-quality face video generation model requires a well-performing pre-trained backbone, a key obstacle that universal models fail to adequately address. Most existing works rely on universal video or image generation models and optimize control mechanisms, but they neglect the evident upper bound in video quality due to the limited capabilities of the backbones, which is a result of the lack of high-quality human face video datasets. In this work, we investigate the unsatisfactory results from related studies, gather and trim existing public talking face video datasets, and additionally collect and annotate a large-scale dataset, resulting in a comprehensive, high-quality multiracial face collection named \textbf{FaceVid-1K}. Using this dataset, we craft several effective pre-trained backbone models for face video generation. Specifically, we conduct experiments with several well-established video generation models, including text-to-video, image-to-video, and unconditional video generation, under various settings. We obtain the corresponding performance benchmarks and compared them with those trained on public datasets to demonstrate the superiority of our dataset. These experiments also allow us to investigate empirical strategies for crafting domain-specific video generation tasks with cost-effective settings. We will make our curated dataset, along with the pre-trained talking face video generation models, publicly available as a resource contribution to hopefully advance the research field.

Related papers

• Movie Gen: A Cast of Media Foundation Models [133.41504332082667]
  We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image.
  arXiv  Detail & Related papers  (2024-10-17T16:22:46Z)
• VidGen-1M: A Large-Scale Dataset for Text-to-video Generation [9.726156628112198]
  We present VidGen-1M, a superior training dataset for text-to-video models. This dataset guarantees high-quality videos and detailed captions with excellent temporal consistency. When used to train the video generation model, this dataset has led to experimental results that surpass those obtained with other models.
  arXiv  Detail & Related papers  (2024-08-05T16:53:23Z)
• VideoClusterNet: Self-Supervised and Adaptive Face Clustering For Videos [2.0719478063181027]
  Video Face Clustering aims to group together detected video face tracks with common facial identities. This problem is very challenging due to the large range of pose, expression, appearance, and lighting variations of a given face across video frames. We present a novel video face clustering approach that learns to adapt a generic face ID model to new video face tracks in a fully self-supervised fashion.
  arXiv  Detail & Related papers  (2024-07-16T23:34:55Z)
• WildVidFit: Video Virtual Try-On in the Wild via Image-Based Controlled Diffusion Models [132.77237314239025]
  Video virtual try-on aims to generate realistic sequences that maintain garment identity and adapt to a person's pose and body shape in source videos. Traditional image-based methods, relying on warping and blending, struggle with complex human movements and occlusions. We reconceptualize video try-on as a process of generating videos conditioned on garment descriptions and human motion. Our solution, WildVidFit, employs image-based controlled diffusion models for a streamlined, one-stage approach.
  arXiv  Detail & Related papers  (2024-07-15T11:21:03Z)
• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets [36.95521842177614]
  We present Stable Video Diffusion - a latent video diffusion model for high-resolution, state-of-the-art text-to-video and image-to-video generation. We identify and evaluate three different stages for successful training of video LDMs: text-to-image pretraining, video pretraining, and high-quality video finetuning.
  arXiv  Detail & Related papers  (2023-11-25T22:28:38Z)
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models [70.19437817951673]
  We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Our approach involves generating a diverse and comprehensive list of 700 prompts for text-to-video generation. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmark, in terms of visual qualities, content qualities, motion qualities, and text-video alignment with 17 well-selected objective metrics.
  arXiv  Detail & Related papers  (2023-10-17T17:50:46Z)
• Probabilistic Adaptation of Text-to-Video Models [181.84311524681536]
  Video Adapter is capable of incorporating the broad knowledge and preserving the high fidelity of a large pretrained video model in a task-specific small video model. Video Adapter is able to generate high-quality yet specialized videos on a variety of tasks such as animation, egocentric modeling, and modeling of simulated and real-world robotics data.
  arXiv  Detail & Related papers  (2023-06-02T19:00:17Z)
• Multiface: A Dataset for Neural Face Rendering [108.44505415073579]
  In this work, we present Multiface, a new multi-view, high-resolution human face dataset. We introduce Mugsy, a large scale multi-camera apparatus to capture high-resolution synchronized videos of a facial performance. The goal of Multiface is to close the gap in accessibility to high quality data in the academic community and to enable research in VR telepresence.
  arXiv  Detail & Related papers  (2022-07-22T17:55:39Z)
• Video Generation from Text Employing Latent Path Construction for Temporal Modeling [70.06508219998778]
  Video generation is one of the most challenging tasks in Machine Learning and Computer Vision fields of study. In this paper, we tackle the text to video generation problem, which is a conditional form of video generation. We believe that video generation from natural language sentences will have an important impact on Artificial Intelligence.
  arXiv  Detail & Related papers  (2021-07-29T06:28:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.