Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency

• URL: http://arxiv.org/abs/2605.13047v1
• Date: Wed, 13 May 2026 06:11:47 GMT
• Title: Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency
• Abstract summary: We introduce Counterfactual Semantic Saliency (CSS) to measure the importance of objects.<n>To evaluate AI-human semantic alignment, we tested prominent vision-language models against a human psychophysics baseline.<n>Models exhibit an overreliance (relative to humans) on large objects (size bias), objects at the center of the image (center bias) and high saliency objects.
• Score: 3.9792042887828565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating whether large vision-language models (VLMs) align with human perception for high-level semantic scene comprehension remains a challenge. Traditional white-box interpretability methods are inapplicable to closed-source architectures and passive metrics fail to isolate causal features. We introduce Counterfactual Semantic Saliency (CSS). This black-box, model-agnostic framework quantifies the importance of objects by measuring the semantic shift induced by their causal ablation from a scene. To evaluate AI-human semantic alignment, we tested prominent VLMs against a human psychophysics baseline comprising 16,289 valid responses across 307 complex natural scenes and 1,306 high-fidelity counterfactual variants. Our analysis reveals a pervasive scene comprehension gap: models exhibit an overreliance (relative to humans) on large objects (size bias), objects at the center of the image (center bias), and high saliency objects. In contrast, models rely less on people in the scenes than our human participants to describe the images. A model's size bias is a primary driver explaining variations in model-human semantic divergence. Code and data will be available at https://github.com/starsky77/Counterfactual-Semantic-Saliency.

Related papers

• Human-AI Perceptual Alignment by Playing Hues and Cues [2.6369756143132634]
  evaluating Contrastive Vision-Language Models (CVLMs) and humans is typically constrained by traditional benchmarks that overlook fine-grained semantic and cultural nuances.<n>We propose a novel evaluation framework that leverages the gamified, discrete color space of the board game Hues and Cues.<n>We evaluate 162 models across multiple architectural families and pre-training datasets to assess their semantic color grounding.
  arXiv  Detail & Related papers  (2026-08-07T12:00:55Z)
• PhyScene3D: Physically Consistent Interactive 3D Tabletop Scene Generation [75.9028457575434]
  Generating 3D tabletop scenes is a fundamental yet underexplored problem for interactive and generalist robotic learning.<n>Here, we introduce PhyScene3D, a framework that reformulates generation as a Human-Mimetic Constructive Process.<n>Experiments demonstrate that PhyScene3D outperforms state-of-the-art approaches in both semantic accuracy and physical validity.
  arXiv  Detail & Related papers  (2026-06-01T03:56:32Z)
• Contextual inference from single objects in Vision-Language models [10.367669666212473]
  How much scene context a single object carries is a well-studied question in human scene perception.<n>How this capacity is organized in vision-language models (VLMs) remains poorly understood.<n>We investigate this question through a systematic behavioral and mechanistic analysis of contextual inference from single objects.
  arXiv  Detail & Related papers  (2026-03-20T13:24:15Z)
• Evaluating the encoding competence of visual language models using uncommon actions [5.816389980109022]
  UAIT is a new evaluation benchmark designed to test the semantic understanding ability of visual language models (VLMs) in uncommon-sense action scenes.<n>We synthesize high-quality uncommon-sense image-text samples using large language models, few-shot prompt engineering, and text-to-image generation.<n>We evaluate multiple state-of-the-art visual language models and compare them with models based on contrastive learning.
  arXiv  Detail & Related papers  (2026-01-12T17:15:45Z)
• Objectness Similarity: Capturing Object-Level Fidelity in 3D Scene Evaluation [57.48883165305971]
  Existing metrics assess overall image quality, leading to discrepancies with human perception.<n>Inspired by neuropsychological insights, we hypothesize that human recognition of 3D scenes fundamentally involves attention to individual objects.<n>This paper presents Objectness SIMilarity (OSIM), a novel evaluation metric for 3D scenes that explicitly focuses on "objects"
  arXiv  Detail & Related papers  (2025-09-11T04:33:27Z)
• Understanding and evaluating computer vision models through the lens of counterfactuals [2.2819712364325047]
  This thesis develops frameworks that use counterfactuals to explain, audit, and mitigate bias in vision classifiers and generative models.<n>By systematically altering semantically meaningful attributes while holding others fixed, these methods uncover spurious correlations.<n>These contributions show counterfactuals as a unifying lens for interpretability, fairness, and causality in both discriminative and generative models.
  arXiv  Detail & Related papers  (2025-08-28T15:11:49Z)
• Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models [65.82564074712836]
  We introduce DIFfusionHOI, a new HOI detector shedding light on text-to-image diffusion models. We first devise an inversion-based strategy to learn the expression of relation patterns between humans and objects in embedding space. These learned relation embeddings then serve as textual prompts, to steer diffusion models generate images that depict specific interactions.
  arXiv  Detail & Related papers  (2024-10-26T12:00:33Z)
• Evaluating Multiview Object Consistency in Humans and Image Models [68.36073530804296]
  We leverage an experimental design from the cognitive sciences which requires zero-shot visual inferences about object shape. We collect 35K trials of behavioral data from over 500 participants. We then evaluate the performance of common vision models.
  arXiv  Detail & Related papers  (2024-09-09T17:59:13Z)
• Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption [64.07607726562841]
  Existing multi-person human reconstruction approaches mainly focus on recovering accurate poses or avoiding penetration. In this work, we tackle the task of reconstructing closely interactive humans from a monocular video. We propose to leverage knowledge from proxemic behavior and physics to compensate the lack of visual information.
  arXiv  Detail & Related papers  (2024-04-17T11:55:45Z)
• An Analysis of Human Alignment of Latent Diffusion Models [4.301861805545143]
  Diffusion models, trained on large amounts of data, showed remarkable performance for image synthesis. They have high error consistency with humans and low texture bias when used for classification. We analyze how well such representations are aligned to human responses on a triplet odd-one-out task.
  arXiv  Detail & Related papers  (2024-03-13T12:31:08Z)
• Attack to Fool and Explain Deep Networks [59.97135687719244]
  We counter-argue by providing evidence of human-meaningful patterns in adversarial perturbations. Our major contribution is a novel pragmatic adversarial attack that is subsequently transformed into a tool to interpret the visual models.
  arXiv  Detail & Related papers  (2021-06-20T03:07:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.