Human-AI Perceptual Alignment by Playing Hues and Cues

• URL: http://arxiv.org/abs/2608.07141v1
• Date: Fri, 07 Aug 2026 12:00:55 GMT
• Title: Human-AI Perceptual Alignment by Playing Hues and Cues
• Abstract summary: evaluating Contrastive Vision-Language Models (CVLMs) and humans is typically constrained by traditional benchmarks that overlook fine-grained semantic and cultural nuances.<n>We propose a novel evaluation framework that leverages the gamified, discrete color space of the board game Hues and Cues.<n>We evaluate 162 models across multiple architectural families and pre-training datasets to assess their semantic color grounding.
• Score: 2.6369756143132634
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating the perceptual alignment between Contrastive Vision-Language Models (CVLMs) and humans is typically constrained by traditional benchmarks that overlook fine-grained semantic and cultural nuances. In this work, we propose a novel evaluation framework that leverages the gamified, discrete color space of the board game Hues and Cues. By mapping the board's 480 color cells to the CIE xy chromaticity diagram, we calculate empirical perceptual distances across a carefully curated 100-word vocabulary spanning seven semantic categories. To properly contextualize model performance, we establish an empirical lower bound of expected error-the Human Consistency baseline-calculated via Leave-One-Out (LOO) cross-validation on a dense dataset of color associations collected from 325 human observers through a custom digital interface. We evaluate 162 models across multiple architectural families and pre-training datasets to assess their semantic color grounding. Our results demonstrate that while CVLMs successfully replicate human cognitive biases, such as idealized memory colors for concrete physical referents (e.g., food and plants), they systematically diverge from the human baseline in abstract, subjective, and pop-culture domains. We identify two distinct failure modes in severely misaligned concepts: semantic misclassification and a systematic uncertainty collapse into a default blue coordinate. Furthermore, we reveal that highly curated pre-training datasets are significantly more effective than massive, uncurated corpora in mitigating these severe misalignments. Ultimately, this work highlights that despite their broad categorization capabilities, current CVLMs still fail to capture the nuanced, localized consensus of human color memory, emphasizing the value of gamified tasks in exposing underlying model biases. The data and code are publicly available to test other metrics.

Related papers

• Do Vision Encoders Exhibit Human-like Color Thresholds? [44.97183276701248]
  In recent years, deep neural networks have become the standard networks for computer vision tasks.<n>Despite the widespread use of deep vision encoders, few studies have investigated whether their internal representations exhibit human-like discrimination thresholds.<n>We present a large-scale exploratory study probing the chromatic sensitivity of more than 50 pretrained vision encoders.
  arXiv  Detail & Related papers  (2026-07-17T22:30:12Z)
• Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency [3.9792042887828565]
  We introduce Counterfactual Semantic Saliency (CSS) to measure the importance of objects.<n>To evaluate AI-human semantic alignment, we tested prominent vision-language models against a human psychophysics baseline.<n>Models exhibit an overreliance (relative to humans) on large objects (size bias), objects at the center of the image (center bias) and high saliency objects.
  arXiv  Detail & Related papers  (2026-05-13T06:11:47Z)
• Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment [48.882448108261826]
  Infrared-Visible image fusion (IVIF) aims to integrate thermal information and detailed spatial structures into a single fused image to enhance perception.<n>Recent IVIF reward-modelling efforts learn from human ratings but use scalar regression on aggregated scores.<n>We introduce FuScore, which utilizes an MLLM to mimic human visual perception by producing continuous quality score.
  arXiv  Detail & Related papers  (2026-05-07T21:38:09Z)
• Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models [1.4969391620741985]
  Vision-language models (VLMs) achieve strong benchmark results, yet can exhibit systematic perceptual weaknesses.<n>We study this gap using Spatial Colour Mixing, a family of colour distortions that overlays structured patterns onto natural images.<n>We show that a simple human-inspired preprocessing step recovers a meaningful portion of performance for several distortion types.
  arXiv  Detail & Related papers  (2026-03-06T10:50:04Z)
• Bridging Human Evaluation to Infrared and Visible Image Fusion [54.71406895277533]
  Infrared and visible image fusion (IVIF) integrates complementary modalities to enhance scene perception.<n>Current methods predominantly focus on optimizing handcrafted losses and objective metrics, often resulting in fusion outcomes that do not align with human visual preferences.<n>We propose a feedback reinforcement framework that bridges human evaluation to infrared and visible image fusion.
  arXiv  Detail & Related papers  (2026-03-04T09:23:57Z)
• Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models [81.45743826739054]
  A major barrier has been the lack of demographic annotations in web-scale datasets such as LAION-400M.<n>We create person-centric annotations for the full dataset, including over 276 million bounding boxes, perceived gender and race/ethnicity labels, and automatically generated captions.<n>Using them, we uncover demographic imbalances and harmful associations, such as the disproportionate linking of men and individuals perceived as Black or Middle Eastern with crime-related and negative content.
  arXiv  Detail & Related papers  (2025-10-04T07:51:59Z)
• COLIBRI Fuzzy Model: Color Linguistic-Based Representation and Interpretation [0.0]
  This paper introduces the Human Perception-Based Fuzzy Color Model, COLIBRI, to bridge the gap between computational color representations and human visual perception.<n>The proposed model uses fuzzy sets and logic to create a framework for color categorization.<n>Our findings are significant for fields such as design, artificial intelligence, marketing, and human-computer interaction.
  arXiv  Detail & Related papers  (2025-07-15T17:01:45Z)
• Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs [65.93003087656754]
  VisFactor is a benchmark that digitizes 20 vision-centric subtests from a well-established cognitive psychology assessment.<n>We evaluate 20 frontier Multimodal Large Language Models (MLLMs) from GPT, Gemini, Claude, LLaMA, Qwen, and SEED families.<n>The best-performing model achieves a score of only 25.19 out of 100, with consistent failures on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination.
  arXiv  Detail & Related papers  (2025-02-23T04:21:32Z)
• Dataset Scale and Societal Consistency Mediate Facial Impression Bias in Vision-Language AI [17.101569078791492]
  We study 43 CLIP vision-language models to determine whether they learn human-like facial impression biases. We show for the first time that the the degree to which a bias is shared across a society predicts the degree to which it is reflected in a CLIP model.
  arXiv  Detail & Related papers  (2024-08-04T08:26:58Z)
• ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection [101.56529337489417]
  We consider the problem of Human-Object Interaction (HOI) Detection, which aims to locate and recognize HOI instances in the form of human, action, object> in images. We argue that multi-level consistencies among objects, actions and interactions are strong cues for generating semantic representations of rare or previously unseen HOIs. Our model takes visual features of candidate human-object pairs and word embeddings of HOI labels as inputs, maps them into visual-semantic joint embedding space and obtains detection results by measuring their similarities.
  arXiv  Detail & Related papers  (2020-08-14T09:11:18Z)
• Appearance Consensus Driven Self-Supervised Human Mesh Recovery [67.20942777949793]
  We present a self-supervised human mesh recovery framework to infer human pose and shape from monocular images. We achieve state-of-the-art results on the standard model-based 3D pose estimation benchmarks. The resulting colored mesh prediction opens up the usage of our framework for a variety of appearance-related tasks beyond the pose and shape estimation.
  arXiv  Detail & Related papers  (2020-08-04T05:40:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.