TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design

• URL: http://arxiv.org/abs/2605.20731v1
• Date: Wed, 20 May 2026 05:27:05 GMT
• Title: TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design
• Abstract summary: We release TASTE (Typography, Aesthetics, Spatial, Tone, Etc.): ten professional designers ranked outputs from four current text-to-image models on nine criteria.<n> TASTE places designer agreement on graphic design between food and movie preferences and photo-style image quality.<n>No pre-trained system in our benchmark, including six open-weight VLM judges from 3B to 33B parameters.
• Score: 43.31865418601155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image models produce graphic design at production scale, but their supervision comes from photo-style preference data with a single overall verdict per comparison. Designers evaluate along several distinct axes, including typography, visual hierarchy, color harmony, layout, and brief fidelity, and a single label collapses them. We release TASTE (Typography, Aesthetics, Spatial, Tone, Etc.): ten professional designers ranked outputs from four current text-to-image models on nine criteria across two disjoint cohorts, yielding 1,600 ratings per criterion plus per-image hallucination flags on the holistic-preference cohorts. We pair the dataset with three contributions. First, a criterion-agnostic signal test framework, using Kendall's tau, majority probability, and Condorcet cycles against exact iid-uniform nulls at p = 4 and R = 5, places designer agreement on graphic design between food and movie preferences and photo-style image quality, with every TASTE criterion rejecting the random-rater null. Second, no pre-trained system in our benchmark, including six open-weight VLM judges from 3B to 33B parameters and three dedicated T2I scorers, HPSv2.1, PickScore-v1, and LAION-Aesthetic-V2, exceeds 0.55 macro agreement with the 5-designer majority; VLM judges trade off position bias against content sensitivity, so scaling moves along this frontier without improving accuracy. Third, a small pairwise-difference head trained on TASTE reaches 0.611, closing roughly half the gap to the 0.741 single-rater ceiling.

Related papers

• Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist [69.81453995715673]
  Existing faithfulness benchmarks rely on simple atomic instructions.<n>We introduce Arena-T2I Hard, a 310-prompt stress benchmark drawn from real arena T2I logs.<n>We propose a dependency-aware checklist reward that decomposes each prompt into a DAG of yes/no questions and zeroes descendants of failed parents.
  arXiv  Detail & Related papers  (2026-06-30T14:17:05Z)
• Zero-Shot Vision-Language Models for Classroom Engagement Recognition: A Benchmark Study of Prompt Sensitivity and Cross-Dataset Generalization [0.44787896002954924]
  We present a systematic benchmark that evaluates five widely-used Vision-Language Models (VLMs)<n>Our experiments reveal three primary failure modes of zero-shot VLMs for engagement recognition.
  arXiv  Detail & Related papers  (2026-06-20T03:53:40Z)
• Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation [52.930280794143876]
  We introduce Qwen-Image-Bench, a creator-centric benchmark co-designed with professional artists and grounded in real-world creation scenarios.<n>We organize these five pillars into a top-down hierarchical taxonomy that decomposes into 23 second-level sub-capabilities and 56 third-level rubrics.<n>We train a unified judge model Q-Judger based on Qwen3.6-27B, supervised by 80 professional annotators from global art academies under blind labeling and triple-review protocols.
  arXiv  Detail & Related papers  (2026-05-27T07:46:43Z)
• Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation [24.67838359287715]
  We introduce PPaint, a matched dual-protocol benchmark in which 15 domain experts, 5 per category, annotate 150 Chinese paintings.<n>We collect 45,900 pairwise expert judgments through a locally dense preference design alongside the matched ratings.<n>Fusing both signals via two independent preference-to-score methods yields a fused expert ground truth.
  arXiv  Detail & Related papers  (2026-05-19T12:44:01Z)
• Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty? [59.923111838399144]
  We introduce the Visual Aesthetic Benchmark (VAB), which casts aesthetic evaluation as comparative selection over candidate sets with matched subject matter.<n>VAB contains 400 tasks and 1,195 images across fine art, photography, and illustration, with labels derived from the consensus of 10 independent expert judges per task.<n>We find that the strongest system identifies both the best and the worst image correctly across three random permutations of the candidate order in only 26.5% of tasks, far below the 68.9% achieved by human experts.
  arXiv  Detail & Related papers  (2026-05-12T19:33:28Z)
• DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation [23.876231579797803]
  We introduce DesignSense-10k, a large-scale dataset of 10,235 human-annotated preference pairs for graphic layout evaluation.<n>We use semantic grouping, layout prediction, filtering, clustering, and VLM-based refinement to produce high-quality comparison pairs.<n>Our analysis shows that frontier VLMs remain unreliable overall and fail catastrophically on the full four-class task.
  arXiv  Detail & Related papers  (2026-02-26T19:03:05Z)
• ImagenHub: Standardizing the evaluation of conditional image generation models [48.51117156168]
  This paper proposes ImagenHub, which is a one-stop library to standardize the inference and evaluation of all conditional image generation models. We design two human evaluation scores, i.e. Semantic Consistency and Perceptual Quality, along with comprehensive guidelines to evaluate generated images. Our human evaluation achieves a high inter-worker agreement of Krippendorff's alpha on 76% models with a value higher than 0.4.
  arXiv  Detail & Related papers  (2023-10-02T19:41:42Z)
• Noisy-Correspondence Learning for Text-to-Image Person Re-identification [50.07634676709067]
  We propose a novel Robust Dual Embedding method (RDE) to learn robust visual-semantic associations even with noisy correspondences. Our method achieves state-of-the-art results both with and without synthetic noisy correspondences on three datasets.
  arXiv  Detail & Related papers  (2023-08-19T05:34:13Z)
• Mix-Teaching: A Simple, Unified and Effective Semi-Supervised Learning Framework for Monocular 3D Object Detection [22.074959519526605]
  Mix-Teaching is an effective semi-supervised learning framework applicable to employ both labeled and unlabeled images in training stage. Mix-Teaching consistently improves MonoFlex and GUPNet by significant margins under various labeling ratios on KITTI dataset.
  arXiv  Detail & Related papers  (2022-07-10T12:07:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.