Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2606.29689v1
• Date: Mon, 29 Jun 2026 01:31:28 GMT
• Title: Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models
• Abstract summary: Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs)<n>We score five open-weight MLLMs against multiple ranked human critiques per photo with reference-based similarity metrics.<n>Stricter lexical and learned metrics show only weak alignment with human critiques.<n>We argue that reference-based similarity rewards a fluent, comprehensive critique style rather than the selectivity and specificity of human critique.
• Score: 16.0895345187708
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): unlike multiple-choice aesthetic benchmarks, it has no single correct answer, and most aesthetic evaluation has measured models against numeric scores rather than the written critiques people actually give. We evaluate MLLM critiques against ranked human references and ask whether they are close to human ones. Using the Reddit Photo Critique Dataset, we score five open-weight MLLMs against multiple ranked human critiques per photo with reference-based similarity metrics, under six prompt conditions that disentangle persona framing, aspect hinting, length control, and single- versus multi-pass generation, and add an image-grounding control that feeds each model the wrong photograph. We find that reference-based similarity gives a misleading picture. Stricter lexical and learned metrics show only weak alignment with human critiques, while a coarse embedding cosine reports broad topical overlap that the grounding control traces to a stable house style rather than image-specific observation. Behaviorally, the models diverge from humans in consistent ways the scores do not surface: even under a length cap they write two to three times as much, cover nearly every aesthetic aspect where humans are selective, engage each aspect more uniformly and at greater depth, and repeat themselves across critiques of the same photo where humans vary. We argue that reference-based similarity rewards a fluent, comprehensive critique style rather than the selectivity and specificity of human critique, and discuss implications for evaluating and training open-ended multimodal generation.

Related papers

• PersonaShot: Benchmarking Person-Centric Narrative Continuity in Multi-Shot Video Generation [118.49150310327475]
  PersonaShot is the first person-centric benchmark for narrative continuity in multi-shot video generation.<n>It contains approximately 1,000 multi-shot segments and 16 metrics spanning physical continuity, affective dynamics, and cinematic grammar.
  arXiv  Detail & Related papers  (2026-08-17T15:34:54Z)
• Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans [50.71222402701153]
  We compare three general-purpose large language models (MLLMs) with human eye-movement scanpaths on goal-directed search (COCO-Search18)<n>All three share one signature: low-entropy, large-amplitude, self-consistent scanpaths that agree with themselves far more closely than two humans agree with each other.<n>The gap sits on a process axis that answer-alignment and saliency metrics do not measure. Because they miss it, such metrics cannot certify human-like vision, and zero-shot models suit outcome and spatial questions but not temporal, process-level
  arXiv  Detail & Related papers  (2026-08-17T12:54:39Z)
• Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency [3.9792042887828565]
  We introduce Counterfactual Semantic Saliency (CSS) to measure the importance of objects.<n>To evaluate AI-human semantic alignment, we tested prominent vision-language models against a human psychophysics baseline.<n>Models exhibit an overreliance (relative to humans) on large objects (size bias), objects at the center of the image (center bias) and high saliency objects.
  arXiv  Detail & Related papers  (2026-05-13T06:11:47Z)
• MORABLES: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables [50.29407048003165]
  We present MORABLES, a human-verified benchmark built from fables and short stories drawn from historical literature.<n>The main task is structured as multiple-choice questions targeting moral inference, with carefully crafted distractors that challenge models to go beyond shallow, extractive question answering.<n>Our findings show that, while larger models outperform smaller ones, they remain susceptible to adversarial manipulation and often rely on superficial patterns rather than true moral reasoning.
  arXiv  Detail & Related papers  (2025-09-15T19:06:10Z)
• CriticAL: Critic Automation with Language Models [31.1575961776287]
  CriticAL generates summary statistics that capture discrepancies between model predictions and data. CriticAL reliably generates correct critiques without hallucinating incorrect ones.
  arXiv  Detail & Related papers  (2024-11-10T20:41:35Z)
• VHELM: A Holistic Evaluation of Vision Language Models [75.88987277686914]
  We present the Holistic Evaluation of Vision Language Models (VHELM) VHELM aggregates various datasets to cover one or more of the 9 aspects: visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety. Our framework is designed to be lightweight and automatic so that evaluation runs are cheap and fast.
  arXiv  Detail & Related papers  (2024-10-09T17:46:34Z)
• CLAIR: Evaluating Image Captions with Large Language Models [69.46906537973518]
  We propose CLAIR, a novel method to evaluate machine-generated image captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Clair provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score.
  arXiv  Detail & Related papers  (2023-10-19T17:59:01Z)
• Privacy Assessment on Reconstructed Images: Are Existing Evaluation Metrics Faithful to Human Perception? [86.58989831070426]
  We study the faithfulness of hand-crafted metrics to human perception of privacy information from reconstructed images. We propose a learning-based measure called SemSim to evaluate the Semantic Similarity between the original and reconstructed images.
  arXiv  Detail & Related papers  (2023-09-22T17:58:04Z)
• Understanding Aesthetics with Language: A Photo Critique Dataset for Aesthetic Assessment [6.201485014848172]
  We propose the Critique Photo Reddit dataset (RPCD), which contains 74K images and 220K comments. We exploit the polarity of the sentiment of criticism as an indicator of aesthetic judgment.
  arXiv  Detail & Related papers  (2022-06-17T08:16:20Z)
• Self-critiquing models for assisting human evaluators [11.1006983438712]
  We fine-tune large language models to write natural language critiques (natural language critical comments) using behavioral cloning. On a topic-based summarization task, critiques written by our models help humans find flaws in summaries that they would have otherwise missed. Larger models write more helpful critiques, and on most tasks, are better at self-critiquing, despite having harder-to-critique outputs.
  arXiv  Detail & Related papers  (2022-06-12T17:40:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.