Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?

• URL: http://arxiv.org/abs/2605.12684v1
• Date: Tue, 12 May 2026 19:33:28 GMT
• Title: Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
• Abstract summary: We introduce the Visual Aesthetic Benchmark (VAB), which casts aesthetic evaluation as comparative selection over candidate sets with matched subject matter.<n>VAB contains 400 tasks and 1,195 images across fine art, photography, and illustration, with labels derived from the consensus of 10 independent expert judges per task.<n>We find that the strongest system identifies both the best and the worst image correctly across three random permutations of the candidate order in only 26.5% of tasks, far below the 68.9% achieved by human experts.
• Score: 59.923111838399144
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multimodal large language models (MLLMs) are now routinely deployed for visual understanding, generation, and curation. A substantial fraction of these applications require an explicit aesthetic judgment. Most existing solutions reduce this judgment to predicting a scalar score for a single image. We first ask whether such scores faithfully capture comparative preference: in a controlled study with eight expert annotators, score-derived rankings align poorly with the same annotators' direct comparisons, while direct ranking yields substantially higher inter-annotator agreement on best- and worst-image labels. Motivated by this finding, we introduce the Visual Aesthetic Benchmark (VAB), which casts aesthetic evaluation as comparative selection over candidate sets with matched subject matter. VAB contains 400 tasks and 1,195 images across fine art, photography, and illustration, with labels derived from the consensus of 10 independent expert judges per task. Evaluating 20 frontier MLLMs and six dedicated visual-quality reward models, we find that the strongest system identifies both the best and the worst image correctly across three random permutations of the candidate order in only 26.5% of tasks, far below the 68.9% achieved by human experts. Fine-tuning a 35B-parameter model on 2,000 expert examples brings its accuracy close to that of a 397B-parameter open-weight model, suggesting that the comparative signal in VAB is transferable. Together, these results expose a clear and measurable gap between current multimodal models and expert aesthetic judgment, and VAB provides the first set-based, expert-grounded testbed on which that gap can be tracked and closed.

Related papers

• VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection [0.04736448323490553]
  VendorBench-100 is a cross-paradigm benchmark for deepfake image detection.<n>It evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework.<n>Our evaluation shows that commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors.
  arXiv  Detail & Related papers  (2026-07-07T13:22:00Z)
• MIBE: Multi-subject Interaction Benchmark and Evaluator for Personalized Image Generation [12.891921839026432]
  We introduce a unified framework comprising a Multi-subject Interaction Benchmark (MIB) and a Multi-subject Interaction Evaluator (MIE)<n>MIB systematically covers diverse relation types and scene complexities through a decoupled data regime.<n>MIE exhibits strong cross-generator generalization on the Gold Set, achieving 0.922 overall pairwise accuracy against human preference.
  arXiv  Detail & Related papers  (2026-07-01T18:44:01Z)
• Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist [69.81453995715673]
  Existing faithfulness benchmarks rely on simple atomic instructions.<n>We introduce Arena-T2I Hard, a 310-prompt stress benchmark drawn from real arena T2I logs.<n>We propose a dependency-aware checklist reward that decomposes each prompt into a DAG of yes/no questions and zeroes descendants of failed parents.
  arXiv  Detail & Related papers  (2026-06-30T14:17:05Z)
• Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2 [0.0]
  I present a solver for ARC-AGI-2, a few-shot visual reasoning benchmark.<n>It generates diverse candidates independently across text, image, and code channels.<n>Unlike self-consistency or majority voting, it reliably recovers correct minority hypotheses on tasks where the modal answer is wrong.
  arXiv  Detail & Related papers  (2026-06-30T11:55:33Z)
• DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues [54.31138496553705]
  DiCoBench is a comprehensive, multi-image high-resolution benchmark for fine-grained perception.<n>Our evaluation reveals a striking performance gap compared to human accuracy (98.3%)<n>We believe DiCoBench will serve as a challenging testbed to drive future research in autonomous, high-resolution multi-image perception.
  arXiv  Detail & Related papers  (2026-06-25T05:02:38Z)
• Medical thinking with multiple images [44.04557445622649]
  We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images.<n>The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case.<n>On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy.
  arXiv  Detail & Related papers  (2026-04-14T18:51:07Z)
• A Sanity Check on Composed Image Retrieval [91.95275287747499]
  Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the desired modification.<n>We introduce FISD, a Fully-Informed Semantically-Diverse benchmark, which employs generative models to precisely control the variables of reference-target image pairs.<n>We propose an automatic multi-round agentic evaluation framework to probe the potential of the existing models in the interactive scenarios.
  arXiv  Detail & Related papers  (2026-04-14T15:52:22Z)
• NTIRE 2026 The 3rd Restore Any Image Model (RAIM) Challenge: Professional Image Quality Assessment (Track 1) [115.35873876159131]
  We present an overview of the NTIRE 2026 challenge on the 3rd Restore Any Image Model in the Wild, specifically focusing on Track 1.<n>In total, the challenge attracted nearly 200 registrations and over 2,500 submissions.<n>The top-performing methods significantly advanced the state of the art in professional IQA.
  arXiv  Detail & Related papers  (2026-04-14T09:44:35Z)
• VB: Visibility Benchmark for Visibility and Perspective Reasoning in Images [0.0]
  We present VB, a benchmark that tests whether vision-language models can determine what is and is not visible in a photograph.<n>Items are organized into 100 families using a 2x2 design that crosses a minimal image edit with a minimal text edit.<n>We score models on confidence-aware accuracy with abstention (CAA), minimal-edit flip rate (MEFR), confidence-ranked selective prediction (SelRank), and second-order perspective reasoning.
  arXiv  Detail & Related papers  (2026-03-03T23:03:11Z)
• How well are open sourced AI-generated image detection models out-of-the-box: A comprehensive benchmark study [5.740397289924559]
  No universal winner exists, with detector rankings exhibiting substantial instability.<n>Our findings challenge the one-size-fits-all'' detector paradigm.
  arXiv  Detail & Related papers  (2026-02-08T04:36:13Z)
• Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image [58.14192385042352]
  We introduce Multimodal RewardBench 2 (MMRB2), the first benchmark for reward models on multimodal understanding and (interleaved) generation.<n>MMRB2 spans four tasks: text-to-image, image editing, interleaved generation, and multimodal reasoning.<n>It provides 1,000 expert-annotated preference pairs per task from 23 models and agents across 21 source tasks.
  arXiv  Detail & Related papers  (2025-12-18T18:56:04Z)
• FLIP Reasoning Challenge [20.706469085872516]
  This paper introduces the FLIP dataset, a benchmark for evaluating AI reasoning capabilities based on human verification tasks.<n>FLIP challenges present users with two orderings of 4 images, requiring them to identify the coherent one.<n>Our experiments evaluate state-of-the-art models, leveraging both vision-language models (VLMs) and large language models (LLMs)
  arXiv  Detail & Related papers  (2025-04-16T17:07:16Z)
• Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare [99.57567498494448]
  We introduce Compare2Score, an all-around LMM-based no-reference IQA model. During training, we generate scaled-up comparative instructions by comparing images from the same IQA dataset. Experiments on nine IQA datasets validate that the Compare2Score effectively bridges text-defined comparative levels during training.
  arXiv  Detail & Related papers  (2024-05-29T17:26:09Z)
• Towards Open-ended Visual Quality Comparison [87.45004129101089]
  We extend the edge of emerging large multi-modality models (LMMs) to advance visual quality comparison into open-ended settings. Co-Instruct is a first-of-its-kind open-source open-ended visual quality comparer. We demonstrate that Co-Instruct achieves in average 30% higher accuracy than state-of-the-art open-source LMMs.
  arXiv  Detail & Related papers  (2024-02-26T15:10:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.