Related papers: The Human Creativity Benchmark

The Human Creativity Benchmark

URL: http://arxiv.org/abs/2606.30561v1
Date: Mon, 29 Jun 2026 16:59:46 GMT
Title: The Human Creativity Benchmark
Abstract summary: In creative domains, professional disagreement reflects genuine differences in taste, not measurement error.<n>We argue that evaluating creative AI requires preserving two distinct signals: convergence and divergence.<n>We present the Human Creativity Benchmark (HCB), a benchmark that operationalizes this separation.
Score: 37.190673682910536
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Modern AI evaluation frameworks treat evaluator disagreement as noise to be resolved. In creative domains, professional disagreement reflects genuine differences in taste, not measurement error. We argue that evaluating creative AI requires preserving two distinct signals: convergence, where professionals align around shared best practices, and divergence, where individual taste legitimately varies. We present the Human Creativity Benchmark (HCB), a benchmark that operationalizes this separation by collecting pairwise preferences, scalar ratings on prompt adherence, usability, and visual appeal, and qualitative rationale from domain professionals. Across 15,000 professional judgments spanning five creative domains and three workflow phases (ideation, mockup, refinement), we find that convergence concentrates on verifiable dimensions like technical correctness and visual hierarchy, while divergence concentrates on taste-driven dimensions like aesthetic direction and conceptual risk. No model excels uniformly across all phases. Collapsing these signals into a single quality metric discards the most actionable information: where models must be correct versus where they should remain steerable.

Related papers

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 12,000 instance-specific rubrics.
arXiv Detail & Related papers (2026-06-26T17:59:15Z)
Do Machines Fail Like Humans? A Human-Centred Out-of-Distribution Spectrum for Mapping Error Alignment [1.974820485832244]
We propose a human-centred framework that redefines the degree of OOD as a spectrum of human perceptual difficulty.<n>We apply this framework to object recognition and reveal unique, regime-dependent model-human alignment rankings and profiles across deep learning architectures.
arXiv Detail & Related papers (2026-03-08T04:51:39Z)
CreativityPrism: A Holistic Benchmark for Large Language Model Creativity [64.18257552903151]
Creativity is often seen as a hallmark of human intelligence.<n>There is still no holistic framework to evaluate their creativity across diverse scenarios.<n>We propose CreativityPrism, an evaluation analysis framework that decomposes creativity into three dimensions: quality, novelty, and diversity.
arXiv Detail & Related papers (2025-10-23T00:22:10Z)
SONA: Learning Conditional, Unconditional, and Mismatching-Aware Discriminator [54.562217603802075]
We introduce Sum of Naturalness and Alignment (SONA), which employs separate projections for naturalness (authenticity) and alignment in the final layer with an inductive bias.<n>Experiments on class-conditional generation tasks show thatSONA achieves superior sample quality and conditional alignment compared to state-of-the-art methods.
arXiv Detail & Related papers (2025-10-06T08:26:06Z)
Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgment [4.334576480811837]
We propose a novel curiosity-driven LLM-as-a-judge for evaluating creative writing.<n>Our method is especially useful in subjective evaluations where not all the annotators agree with each other.
arXiv Detail & Related papers (2025-10-01T04:29:36Z)
RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark [71.3555284685426]
We introduce RealUnify, a benchmark designed to evaluate bidirectional capability synergy.<n>RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks.<n>We find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient.
arXiv Detail & Related papers (2025-09-29T15:07:28Z)
Modeling Beyond MOS: Quality Assessment Models Must Integrate Context, Reasoning, and Multimodality [45.34252727738116]
Mean Opinion Score (MOS) is no longer sufficient as the sole supervisory signal for multimedia quality assessment models.<n>By reframing quality assessment as a contextual, explainable, and multimodal modeling task, we aim to catalyze a shift toward more robust, human-aligned, and trustworthy evaluation systems.
arXiv Detail & Related papers (2025-05-26T08:52:02Z)
Democratizing Reward Design for Personal and Representative Value-Alignment [10.1630183955549]
We introduce Interactive-Reflective Dialogue Alignment, a method that iteratively engages users in reflecting on and specifying their subjective value definitions. This system learns individual value definitions through language-model-based preference elicitation and constructs personalized reward models. Our findings demonstrate diverse definitions of value-aligned behaviour and show that our system can accurately capture each person's unique understanding.
arXiv Detail & Related papers (2024-10-29T16:37:01Z)
Exploring the Trade-off between Plausibility, Change Intensity and Adversarial Power in Counterfactual Explanations using Multi-objective Optimization [73.89239820192894]
We argue that automated counterfactual generation should regard several aspects of the produced adversarial instances. We present a novel framework for the generation of counterfactual examples.
arXiv Detail & Related papers (2022-05-20T15:02:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.