Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models

• URL: http://arxiv.org/abs/2603.06141v1
• Date: Fri, 06 Mar 2026 10:50:04 GMT
• Title: Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models
• Abstract summary: Vision-language models (VLMs) achieve strong benchmark results, yet can exhibit systematic perceptual weaknesses.<n>We study this gap using Spatial Colour Mixing, a family of colour distortions that overlays structured patterns onto natural images.<n>We show that a simple human-inspired preprocessing step recovers a meaningful portion of performance for several distortion types.
• Score: 1.4969391620741985
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision-language models (VLMs) achieve strong benchmark results, yet can exhibit systematic perceptual weaknesses: structured, large changes to pixel values can cause confident yet nonsensical predictions, even when the underlying scene remains easily recognizable to humans. We study this gap using Spatial Colour Mixing, a programmatic family of colour distortions that overlays structured patterns (in both RGB and Ostwald colour systems) onto natural images. We introduce a framework of eight spatial colour mixing variants and evaluate nine VLMs across three model families on four datasets. Across models and datasets, accuracy degrades sharply with increasing distortion, and scaling the language model does not reliably mitigate the failure. In a human study with 61 participants on an animal recognition dataset, humans substantially outperform VLMs under the same distortions. Finally, we show that a simple human-inspired preprocessing step recovers a meaningful portion of performance for several distortion types, motivating perception-aware preprocessing and tool-use as practical strategies for improving VLM robustness.

Related papers

• Human-AI Perceptual Alignment by Playing Hues and Cues [2.6369756143132634]
  evaluating Contrastive Vision-Language Models (CVLMs) and humans is typically constrained by traditional benchmarks that overlook fine-grained semantic and cultural nuances.<n>We propose a novel evaluation framework that leverages the gamified, discrete color space of the board game Hues and Cues.<n>We evaluate 162 models across multiple architectural families and pre-training datasets to assess their semantic color grounding.
  arXiv  Detail & Related papers  (2026-08-07T12:00:55Z)
• Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks [11.064940886724257]
  This paper presents a systematic evaluation framework for vision-language models (VLMs) under natural adversarial scenarios.<n>We measure the natural adversarial performance of selected VLMs for zero-shot image classification, semantic segmentation, and visual question answering.<n>Our analysis reveals that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models significantly reduce performance for natural language-induced adversarial examples.
  arXiv  Detail & Related papers  (2026-04-06T06:48:32Z)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models [64.56065206447788]
  Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets.<n>We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometrics.<n>We show that low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions.
  arXiv  Detail & Related papers  (2026-03-06T10:58:02Z)
• LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models [49.92148175114169]
  We perform a systematic vulnerability analysis by introducing controlled perturbations across seven dimensions.<n>Models exhibit extreme sensitivity to perturbation factors, including camera viewpoints and robot initial states.<n>Surprisingly, models are largely insensitive to language variations, with further experiments revealing that models tend to ignore language instructions completely.
  arXiv  Detail & Related papers  (2025-10-15T14:51:36Z)
• ColorBlindnessEval: Can Vision-Language Models Pass Color Blindness Tests? [3.7113284199760552]
  This paper presents ColorBlindnessEval, a novel benchmark designed to evaluate the robustness of Vision-Language Models (VLMs) in visually adversarial scenarios.<n>Our dataset comprises 500 Ishihara-like images featuring numbers from 0 to 99 with varying color combinations.<n>We assess 9 VLMs using Yes/No and open-ended prompts and compare their performance with human participants.
  arXiv  Detail & Related papers  (2025-09-23T14:33:21Z)
• Visual Bias and Interpretability in Deep Learning for Dermatological Image Analysis [0.0]
  This study proposes a deep learning framework for multi-class skin disease classification.<n>We benchmark the performance of pre-trained convolutional neural networks (DenseNet201, Efficient-NetB5) and transformer-based models (ViT, Swin Transformer, DinoV2 Large)<n>Results show that DinoV2 with RGB pre-processing achieves the highest accuracy (up to 93%) and F1-scores across all variants.
  arXiv  Detail & Related papers  (2025-08-06T15:57:49Z)
• Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs [65.93003087656754]
  VisFactor is a benchmark that digitizes 20 vision-centric subtests from a well-established cognitive psychology assessment.<n>We evaluate 20 frontier Multimodal Large Language Models (MLLMs) from GPT, Gemini, Claude, LLaMA, Qwen, and SEED families.<n>The best-performing model achieves a score of only 25.19 out of 100, with consistent failures on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination.
  arXiv  Detail & Related papers  (2025-02-23T04:21:32Z)
• A Value Mapping Virtual Staining Framework for Large-scale Histological Imaging [36.95712533471744]
  We introduce a general virtual staining framework that is adaptable to various conditions.<n>We propose a loss function based on the value mapping constraint to ensure the accuracy of virtual coloring between different pathological modalities.
  arXiv  Detail & Related papers  (2025-01-07T07:45:21Z)
• Exploring Color Invariance through Image-Level Ensemble Learning [7.254270666779331]
  This study introduces a learning strategy named Random Color Erasing. It selectively erases partial or complete color information in the training data without disrupting the original image structure. This approach mitigates the risk of overfitting and enhances the model's ability to handle color variation.
  arXiv  Detail & Related papers  (2024-01-19T06:04:48Z)
• Identifying and Mitigating Model Failures through Few-shot CLIP-aided Diffusion Generation [65.268245109828]
  We propose an end-to-end framework to generate text descriptions of failure modes associated with spurious correlations. These descriptions can be used to generate synthetic data using generative models, such as diffusion models. Our experiments have shown remarkable textbfimprovements in accuracy ($sim textbf21%$) on hard sub-populations.
  arXiv  Detail & Related papers  (2023-12-09T04:43:49Z)
• Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets [46.19529338280716]
  Language models, characterized by their black-box nature, often hallucinate and display sensitivity to input perturbations.<n>We introduce a methodology designed to examine how input perturbations affect language models across various scales.<n>We present three distinct fine-tuning strategies to address robustness against multiple perturbations.
  arXiv  Detail & Related papers  (2023-11-15T02:59:10Z)
• Semantic Image Attack for Visual Model Diagnosis [80.36063332820568]
  In practice, metric analysis on a specific train and test dataset does not guarantee reliable or fair ML models. This paper proposes Semantic Image Attack (SIA), a method based on the adversarial attack that provides semantic adversarial images.
  arXiv  Detail & Related papers  (2023-03-23T03:13:04Z)
• Debiasing Vision-Language Models via Biased Prompts [79.04467131711775]
  We propose a general approach for debiasing vision-language foundation models by projecting out biased directions in the text embedding. We show that debiasing only the text embedding with a calibrated projection matrix suffices to yield robust classifiers and fair generative models.
  arXiv  Detail & Related papers  (2023-01-31T20:09:33Z)
• Neural Descent for Visual 3D Human Pose and Shape [67.01050349629053]
  We present deep neural network methodology to reconstruct the 3d pose and shape of people, given an input RGB image. We rely on a recently introduced, expressivefull body statistical 3d human model, GHUM, trained end-to-end. Central to our methodology, is a learning to learn and optimize approach, referred to as HUmanNeural Descent (HUND), which avoids both second-order differentiation.
  arXiv  Detail & Related papers  (2020-08-16T13:38:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.