Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance

• URL: http://arxiv.org/abs/2604.01848v2
• Date: Fri, 03 Apr 2026 11:47:24 GMT
• Title: Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance
• Abstract summary: This work investigates the fundamental fragility of state-of-the-art Vision-Language Models (VLMs) under basic geometric transformations.<n>While modern VLMs excel at semantic tasks, they exhibit systematic failures at a more fundamental level.
• Score: 4.177067537825386
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work investigates the fundamental fragility of state-of-the-art Vision-Language Models (VLMs) under basic geometric transformations. While modern VLMs excel at semantic tasks such as recognizing objects in canonical orientations and describing complex scenes, they exhibit systematic failures at a more fundamental level: lack of robust spatial invariance and equivariance required to reliably determine object identity under simple rotations, scaling, and identity transformations. We demonstrate this limitation through a systematic evaluation across diverse visual domains, including symbolic sketches, natural photographs, and abstract art. Performance drops sharply as semantic content becomes sparse, and this behavior is observed across architectures, model capacities, and prompting strategies. Overall, our results reveal a systematic gap between semantic understanding and spatial reasoning in current VLMs, highlighting the need for stronger geometric grounding in future multimodal systems.

Related papers

• SpatialImaginer: Towards Adaptive Visual Imagination for Spatial Reasoning [67.67774742200626]
  spatial intelligence, which refers to the ability to reason about geometric and physical structure from visual observations, is a core challenge for large language models.<n>We propose a unified multimodal generation framework that integrates textual reasoning with visual imagination.<n>Our framework adopts a divide-and-conquer strategy, using text chain-of-thought for high-level semantic planning and the visual imagination for geometry-sensitive state transformation and consistency preservation.
  arXiv  Detail & Related papers  (2026-04-19T11:21:59Z)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models [84.94288033791346]
  We conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs.<n>We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation.<n>We propose Predictive Regularization to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM's internal representations.
  arXiv  Detail & Related papers  (2026-03-21T13:10:37Z)
• Revisiting Multi-Task Visual Representation Learning [52.93947931352643]
  We introduce MTV, a principled multi-task visual pretraining framework.<n>We leverage high-capacity "expert" models to synthesize dense, structured pseudo-labels at scale.<n>Our results demonstrate that MTV achieves "best-of-both-worlds" performance.
  arXiv  Detail & Related papers  (2026-01-20T11:59:19Z)
• The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs [44.71703930770065]
  We present The Perceptual Observatory, a framework that characterizes MLLMs across verticals like face matching and text-in-vision comprehension capabilities.<n>The Perceptual Observatory moves beyond leaderboard accuracy to yield insights into how MLLMs preserve perceptual grounding and relational structure under perturbations.
  arXiv  Detail & Related papers  (2025-12-17T20:22:23Z)
• Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling [68.14113731953971]
  This paper introduces MILO, an Implicit spatIaL wOrld modeling paradigm that simulates human-like imagination.<n>We show that our approach significantly enhances spatial reasoning capabilities across multiple baselines and benchmarks.
  arXiv  Detail & Related papers  (2025-12-01T16:01:41Z)
• Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward [1.7971686967440696]
  V$2$R-Bench is a benchmark framework for evaluating Visual Variation Robustness of LVLMs.<n>We show that advanced models that excel at complex vision-language tasks significantly underperform on simple tasks such as object recognition.<n>These vulnerabilities stem from error accumulation in the pipeline architecture and inadequate multimodal alignment.
  arXiv  Detail & Related papers  (2025-04-23T14:01:32Z)
• MATHGLANCE: Multimodal Large Language Models Do Not Know Where to Look in Mathematical Diagrams [65.02628814094639]
  Diagrams serve as a fundamental form of visual language, representing complex concepts and their inter-relationships through structured symbols, shapes, and spatial arrangements.<n>Current benchmarks conflate perceptual and reasoning tasks, making it difficult to assess whether Multimodal Large Language Models genuinely understand mathematical diagrams beyond superficial pattern recognition.<n>We introduce MATHGLANCE, a benchmark specifically designed to isolate and evaluate mathematical perception in MLLMs.<n>We construct GeoPeP, a perception-oriented dataset of 200K structured geometry image-text annotated with geometric primitives and precise spatial relationships.
  arXiv  Detail & Related papers  (2025-03-26T17:30:41Z)
• Hierarchical Invariance for Robust and Interpretable Vision Tasks at Larger Scales [54.78115855552886]
  We show how to construct over-complete invariants with a Convolutional Neural Networks (CNN)-like hierarchical architecture. With the over-completeness, discriminative features w.r.t. the task can be adaptively formed in a Neural Architecture Search (NAS)-like manner. For robust and interpretable vision tasks at larger scales, hierarchical invariant representation can be considered as an effective alternative to traditional CNN and invariants.
  arXiv  Detail & Related papers  (2024-02-23T16:50:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.