MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans

• URL: http://arxiv.org/abs/2506.20879v2
• Date: Tue, 05 Aug 2025 19:44:46 GMT
• Title: MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans
• Authors: Shubhankar Borse, Seokeon Choi, Sunghyun Park, Jeongho Kim, Shreya Kadambi, Risheek Garrepalli, Sungrack Yun, Munawar Hayat, Fatih Porikli,
• Abstract summary: We introduce MultiHuman-Testbench, a novel benchmark for rigorously evaluating generative models for multi-human generation.<n>The benchmark comprises 1800 samples, including carefully curated text prompts, describing a range of simple to complex human actions.<n>We propose a multi-faceted evaluation suite employing four key metrics to quantify face count, ID similarity, prompt alignment, and action detection.
• Score: 60.6183017400517
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generation of images containing multiple humans, performing complex actions, while preserving their facial identities, is a significant challenge. A major factor contributing to this is the lack of a dedicated benchmark. To address this, we introduce MultiHuman-Testbench, a novel benchmark for rigorously evaluating generative models for multi-human generation. The benchmark comprises 1800 samples, including carefully curated text prompts, describing a range of simple to complex human actions. These prompts are matched with a total of 5,550 unique human face images, sampled uniformly to ensure diversity across age, ethnic background, and gender. Alongside captions, we provide human-selected pose conditioning images which accurately match the prompt. We propose a multi-faceted evaluation suite employing four key metrics to quantify face count, ID similarity, prompt alignment, and action detection. We conduct a thorough evaluation of a diverse set of models, including zero-shot approaches and training-based methods, with and without regional priors. We also propose novel techniques to incorporate image and region isolation using human segmentation and Hungarian matching, significantly improving ID similarity. Our proposed benchmark and key findings provide valuable insights and a standardized tool for advancing research in multi-human image generation. The dataset and evaluation codes will be available at https://github.com/Qualcomm-AI-research/MultiHuman-Testbench.

Related papers

• Beyond Walking: A Large-Scale Image-Text Benchmark for Text-based Person Anomaly Search [25.907668574771705]
  We propose a new task, text-based person anomaly search, locating pedestrians engaged in both routine or anomalous activities via text.<n>To enable the training and evaluation of this new task, we construct a large-scale image-text Pedestrian Anomaly Behavior benchmark.<n>Experiments on the proposed benchmark show that synthetic training data facilitates the fine-grained behavior retrieval, and the proposed pose-aware method arrives at 84.93% recall@1 accuracy.
  arXiv  Detail & Related papers  (2024-11-26T09:50:15Z)
• Evaluating Multiview Object Consistency in Humans and Image Models [68.36073530804296]
  We leverage an experimental design from the cognitive sciences which requires zero-shot visual inferences about object shape. We collect 35K trials of behavioral data from over 500 participants. We then evaluate the performance of common vision models.
  arXiv  Detail & Related papers  (2024-09-09T17:59:13Z)
• UniHuman: A Unified Model for Editing Human Images in the Wild [49.896715833075106]
  We propose UniHuman, a unified model that addresses multiple facets of human image editing in real-world settings. To enhance the model's generation quality and generalization capacity, we leverage guidance from human visual encoders. In user studies, UniHuman is preferred by the users in an average of 77% of cases.
  arXiv  Detail & Related papers  (2023-12-22T05:00:30Z)
• The Chosen One: Consistent Characters in Text-to-Image Diffusion Models [71.15152184631951]
  We propose a fully automated solution for consistent character generation with the sole input being a text prompt. Our method strikes a better balance between prompt alignment and identity consistency compared to the baseline methods.
  arXiv  Detail & Related papers  (2023-11-16T18:59:51Z)
• SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation [60.94239810407917]
  This paper presents a multi-purpose algorithm for simultaneous face recognition, facial expression recognition, age estimation, and face attribute estimation based on a single Swin Transformer. To address the conflicts among multiple tasks, a Multi-Level Channel Attention (MLCA) module is integrated into each task-specific analysis. Experiments show that the proposed model has a better understanding of the face and achieves excellent performance for all tasks.
  arXiv  Detail & Related papers  (2023-08-22T15:38:39Z)
• MiVOLO: Multi-input Transformer for Age and Gender Estimation [0.0]
  We present MiVOLO, a straightforward approach for age and gender estimation using the latest vision transformer. Our method integrates both tasks into a unified dual input/output model. We compare our model's age recognition performance with human-level accuracy and demonstrate that it significantly outperforms humans across a majority of age ranges.
  arXiv  Detail & Related papers  (2023-07-10T14:58:10Z)
• ArtWhisperer: A Dataset for Characterizing Human-AI Interactions in Artistic Creations [26.4215586218117]
  This work investigates how people use text-to-image models to generate desired target images. We created ArtWhisperer, an online game where users are given a target image and are tasked with iteratively finding a prompt that creates a similar-looking image as the target. We recorded over 50,000 human-AI interactions; each interaction corresponds to one text prompt created by a user and the corresponding generated image.
  arXiv  Detail & Related papers  (2023-06-13T21:10:45Z)
• Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark [80.79082788458602]
  We provide a new multi-task benchmark for evaluating text-to-image models. We compare the most common open-source (Stable Diffusion) and commercial (DALL-E 2) models. Twenty computer science AI graduate students evaluated the two models, on three tasks, at three difficulty levels, across ten prompts each.
  arXiv  Detail & Related papers  (2022-11-22T09:27:53Z)
• End-to-End Context-Aided Unicity Matching for Person Re-identification [100.02321122258638]
  We propose an end-to-end person unicity matching architecture for learning and refining the person matching relations. We use the samples' global context relationship to refine the soft matching results and reach the matching unicity through bipartite graph matching. Given full consideration to real-world person re-identification applications, we achieve the unicity matching in both one-shot and multi-shot settings.
  arXiv  Detail & Related papers  (2022-10-20T07:33:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.