HumanRefiner: Benchmarking Abnormal Human Generation and Refining with Coarse-to-fine Pose-Reversible Guidance

• URL: http://arxiv.org/abs/2407.06937v1
• Date: Tue, 9 Jul 2024 15:14:41 GMT
• Title: HumanRefiner: Benchmarking Abnormal Human Generation and Refining with Coarse-to-fine Pose-Reversible Guidance
• Authors: Guian Fang, Wenbiao Yan, Yuanfan Guo, Jianhua Han, Zutao Jiang, Hang Xu, Shengcai Liao, Xiaodan Liang,
• Abstract summary: AbHuman is the first large-scale synthesized human benchmark focusing on anatomical anomalies. HumanRefiner is a novel plug-and-play approach for the coarse-to-fine refinement of human anomalies in text-to-image generation.
• Score: 80.97360194728705
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-image diffusion models have significantly advanced in conditional image generation. However, these models usually struggle with accurately rendering images featuring humans, resulting in distorted limbs and other anomalies. This issue primarily stems from the insufficient recognition and evaluation of limb qualities in diffusion models. To address this issue, we introduce AbHuman, the first large-scale synthesized human benchmark focusing on anatomical anomalies. This benchmark consists of 56K synthesized human images, each annotated with detailed, bounding-box level labels identifying 147K human anomalies in 18 different categories. Based on this, the recognition of human anomalies can be established, which in turn enhances image generation through traditional techniques such as negative prompting and guidance. To further boost the improvement, we propose HumanRefiner, a novel plug-and-play approach for the coarse-to-fine refinement of human anomalies in text-to-image generation. Specifically, HumanRefiner utilizes a self-diagnostic procedure to detect and correct issues related to both coarse-grained abnormal human poses and fine-grained anomaly levels, facilitating pose-reversible diffusion generation. Experimental results on the AbHuman benchmark demonstrate that HumanRefiner significantly reduces generative discrepancies, achieving a 2.9x improvement in limb quality compared to the state-of-the-art open-source generator SDXL and a 1.4x improvement over DALL-E 3 in human evaluations. Our data and code are available at https://github.com/Enderfga/HumanRefiner.

Related papers

• Boost Your Own Human Image Generation Model via Direct Preference Optimization with AI Feedback [5.9726297901501475]
  We introduce a novel approach tailored specifically for human image generation utilizing Direct Preference Optimization (DPO) Specifically, we introduce an efficient method for constructing a specialized DPO dataset for training human image generation models without the need for costly human feedback. Our method demonstrates its versatility and effectiveness in generating human images, including personalized text-to-image generation.
  arXiv  Detail & Related papers  (2024-05-30T16:18:05Z)
• AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation [55.179287851188036]
  We introduce a novel all-in-one-stage framework, AiOS, for expressive human pose and shape recovery without an additional human detection step. We first employ a human token to probe a human location in the image and encode global features for each instance. Then, we introduce a joint-related token to probe the human joint in the image and encoder a fine-grained local feature.
  arXiv  Detail & Related papers  (2024-03-26T17:59:23Z)
• Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation [24.49857926071974]
  Vanilla text-to-image diffusion models struggle with generating accurate human images. Existing methods address this issue mostly by fine-tuning the model with extra images or adding additional controls. This paper explores the integration of human-centric priors directly into the model fine-tuning stage.
  arXiv  Detail & Related papers  (2024-03-08T11:59:32Z)
• Towards the Detection of AI-Synthesized Human Face Images [12.090322373964124]
  This paper presents a benchmark including human face images produced by Generative Adversarial Networks (GANs) and a variety of DMs. Then, the forgery traces introduced by different generative models have been analyzed in the frequency domain to draw various insights. The paper further demonstrates that a detector trained with frequency representation can generalize well to other unseen generative models.
  arXiv  Detail & Related papers  (2024-02-13T19:37:44Z)
• HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion [53.1558345421646]
  We propose HumanRef, a 3D human generation framework from a single-view input. To ensure the generated 3D model is photorealistic and consistent with the input image, HumanRef introduces a novel method called reference-guided score distillation sampling. Experimental results demonstrate that HumanRef outperforms state-of-the-art methods in generating 3D clothed humans.
  arXiv  Detail & Related papers  (2023-11-28T17:06:28Z)
• Deceptive-Human: Prompt-to-NeRF 3D Human Generation with 3D-Consistent Synthetic Images [67.31920821192323]
  Deceptive-Human is a novel framework capitalizing state-of-the-art control diffusion models (e.g., ControlNet) to generate a high-quality controllable 3D human NeRF. Our method is versatile and readily accommodating, including a text prompt and additional data such as 3D mesh, poses, and seed images. The resulting 3D human NeRF model empowers the synthesis of highly photorealistic views from 360-degree perspectives.
  arXiv  Detail & Related papers  (2023-11-27T15:49:41Z)
• HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion [114.15397904945185]
  We propose a unified framework, HyperHuman, that generates in-the-wild human images of high realism and diverse layouts. Our model enforces the joint learning of image appearance, spatial relationship, and geometry in a unified network. Our framework yields the state-of-the-art performance, generating hyper-realistic human images under diverse scenarios.
  arXiv  Detail & Related papers  (2023-10-12T17:59:34Z)
• Exploring the Robustness of Human Parsers Towards Common Corruptions [99.89886010550836]
  We construct three corruption robustness benchmarks, termed LIP-C, ATR-C, and Pascal-Person-Part-C, to assist us in evaluating the risk tolerance of human parsing models. Inspired by the data augmentation strategy, we propose a novel heterogeneous augmentation-enhanced mechanism to bolster robustness under commonly corrupted conditions.
  arXiv  Detail & Related papers  (2023-09-02T13:32:14Z)
• Diffusion Models as Artists: Are we Closing the Gap between Humans and Machines? [4.802758600019422]
  We adapt the 'diversity vs. recognizability' scoring framework from Boutin et al, 2022. We find that one-shot diffusion models have indeed started to close the gap between humans and machines.
  arXiv  Detail & Related papers  (2023-01-27T14:08:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.