From Bias to Balance: Exploring and Mitigating Spatial Bias in LVLMs

• URL: http://arxiv.org/abs/2509.21984v1
• Date: Fri, 26 Sep 2025 07:07:03 GMT
• Title: From Bias to Balance: Exploring and Mitigating Spatial Bias in LVLMs
• Authors: Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Weili Guan, Jun Yu, Min Zhang,
• Abstract summary: Large Vision-Language Models (LVLMs) have achieved remarkable success across a wide range of multimodal tasks.<n>We focus on how models respond when identical key visual information is placed at different locations within an image.<n>We introduce Balanced Position Assignment (BaPA), a simple yet effective mechanism that assigns identical position embeddings to all image tokens.
• Score: 57.01486941224062
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable success across a wide range of multimodal tasks, yet their robustness to spatial variations remains insufficiently understood. In this work, we present a systematic study of the spatial bias of LVLMs, focusing on how models respond when identical key visual information is placed at different locations within an image. Through a carefully designed probing dataset, we demonstrate that current LVLMs often produce inconsistent outputs under such spatial shifts, revealing a fundamental limitation in their spatial-semantic understanding. Further analysis shows that this phenomenon originates not from the vision encoder, which reliably perceives and interprets visual content across positions, but from the unbalanced design of position embeddings in the language model component. In particular, the widely adopted position embedding strategies, such as RoPE, introduce imbalance during cross-modal interaction, leading image tokens at different positions to exert unequal influence on semantic understanding. To mitigate this issue, we introduce Balanced Position Assignment (BaPA), a simple yet effective mechanism that assigns identical position embeddings to all image tokens, promoting a more balanced integration of visual information. Extensive experiments show that BaPA enhances the spatial robustness of LVLMs without retraining and further boosts their performance across diverse multimodal benchmarks when combined with lightweight fine-tuning. Further analysis of information flow reveals that BaPA yields balanced attention, enabling more holistic visual understanding.

Related papers

• The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs [44.71703930770065]
  We present The Perceptual Observatory, a framework that characterizes MLLMs across verticals like face matching and text-in-vision comprehension capabilities.<n>The Perceptual Observatory moves beyond leaderboard accuracy to yield insights into how MLLMs preserve perceptual grounding and relational structure under perturbations.
  arXiv  Detail & Related papers  (2025-12-17T20:22:23Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• HERO: Rethinking Visual Token Early Dropping in High-Resolution Large Vision-Language Models [60.028070589466445]
  We propose HERO, a framework that integrates content-adaptive token budget allocation with function-aware token selection.<n>This study provides both empirical insights and practical solutions toward efficient inference in HR-LVLMs.
  arXiv  Detail & Related papers  (2025-09-16T13:22:08Z)
• ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models [68.46716645478661]
  Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and reasoning about visual content.<n>Current VLMs excel primarily at egocentric spatial reasoning (from the camera's perspective) but fail to generalize to allocentric viewpoints.<n>We introduce ViewSpatial-Bench, the first comprehensive benchmark designed specifically for multi-viewpoint spatial localization recognition evaluation.
  arXiv  Detail & Related papers  (2025-05-27T17:59:26Z)
• MLLMs are Deeply Affected by Modality Bias [158.64371871084478]
  Recent advances in Multimodal Large Language Models (MLLMs) have shown promising results in integrating diverse modalities such as texts and images.<n>MLLMs are heavily influenced by modality bias, often relying on language while under-utilizing other modalities like visual inputs.<n>This paper argues that MLLMs are deeply affected by modality bias, highlighting its manifestations across various tasks.
  arXiv  Detail & Related papers  (2025-05-24T11:49:31Z)
• Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward [1.7971686967440696]
  V$2$R-Bench is a benchmark framework for evaluating Visual Variation Robustness of LVLMs.<n>We show that advanced models that excel at complex vision-language tasks significantly underperform on simple tasks such as object recognition.<n>These vulnerabilities stem from error accumulation in the pipeline architecture and inadequate multimodal alignment.
  arXiv  Detail & Related papers  (2025-04-23T14:01:32Z)
• Identifying and Mitigating Position Bias of Multi-image Vision-Language Models [8.477985931416303]
  We show that Large Vision-Language Models (LVLMs) struggle to robustly utilize information across multiple images.<n>We propose SoFt Attention (SoFA), a training-free approach that mitigates this bias.<n> Experimental results demonstrate that SoFA reduces position bias and enhances the reasoning performance of existing LVLMs.
  arXiv  Detail & Related papers  (2025-03-18T00:45:02Z)
• Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration [22.39558434131574]
  Large Vision-Language Models (LVLMs) generate responses that are not factually aligned with the visual content.<n>We introduce a training-free solution, Uniform Attention (UAC), that estimates the bias from single meaningless input image.<n>We also introduce a fine-tuning solution, Dynamic Attention (DAC), that enforces the consistent outputs wherever the object locates in the image.
  arXiv  Detail & Related papers  (2025-02-04T03:27:38Z)
• Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge [24.538839144639653]
  Large Vision-Language Models (LVLMs) integrate separately pre-trained vision and language components. These models frequently encounter a core issue of "cognitive misalignment" between the vision encoder (VE) and the large language model (LLM)
  arXiv  Detail & Related papers  (2024-11-25T18:33:14Z)
• Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders [89.41055673919895]
  This study explores the design space for MLLMs using a mixture of vision encoders and resolutions.<n>We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies.<n>The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks.
  arXiv  Detail & Related papers  (2024-08-28T17:59:31Z)
• Debiasing Multimodal Large Language Models via Penalization of Language Priors [38.97645845493758]
  Multimodal Large Language Models (MLLMs) have become indispensable tools in computer vision and natural language processing.<n>Despite their advancements, our investigation reveals a noteworthy bias: the generated content is often driven more by the inherent priors of the underlying Large Language Models (LLMs) than by the input image.<n>We propose two simple, training-free strategies to rectify these biases and redirect the model's focus toward visual information.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.