Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs

• URL: http://arxiv.org/abs/2608.15085v1
• Date: Sat, 15 Aug 2026 07:04:53 GMT
• Title: Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs
• Abstract summary: Multimodal large language models (MLLMs) exhibit substantial performance degradation in non-English visual reasoning.<n>We propose textbfANCHOR, a training framework employing Proactive Visual Anchoring (PVA) to accelerate early visual semantic emergence.<n>Mechanistic interventions confirm that ANCHOR successfully restores the causal influence of visual signals during early translation.
• Score: 17.007513213148698
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language models (MLLMs) exhibit substantial performance degradation in non-English visual reasoning, despite the strong multilingual competence of their text-only backbones. While mechanistic evidence from text-only models suggests that non-English inputs are routed through an English-centric latent space, the multimodal implications of this phenomenon remain unexplored. Through rigorous mechanistic analysis, we identify the \textbf{Ghost Anchor} phenomenon: a temporal modality asynchrony where linguistic translation to the English semantic manifold completes in early layers, while visual semanticization remains immature. Consequently, visual signals are physically present yet functionally invisible during the early alignment window. To rectify this, we propose \textbf{ANCHOR}, a training framework employing Proactive Visual Anchoring (PVA) to accelerate early visual semantic emergence, ensuring visual representations proactively guide linguistic translation. Mechanistic interventions confirm that ANCHOR successfully restores the causal influence of visual signals during early translation. Furthermore, extensive experiments on XMMMU, MaXM, and CVQA demonstrate that ANCHOR consistently outperforms standard baselines, achieving robust visual reasoning across both fine-tuned and zero-shot languages.

Related papers

• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models [84.94288033791346]
  We conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs.<n>We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation.<n>We propose Predictive Regularization to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM's internal representations.
  arXiv  Detail & Related papers  (2026-03-21T13:10:37Z)
• Towards Understanding Multimodal Fine-Tuning: Spatial Features [25.349396112139214]
  Vision-Language Models (VLMs) achieve strong performance on a wide range of tasks by pairing a vision encoder with a pre-trained language model.<n>We present the first mechanistic analysis of VLM adaptation using stage-wise model diffing.
  arXiv  Detail & Related papers  (2026-02-06T18:48:18Z)
• When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training [57.230355403478995]
  We investigate the development of language-agnostic concept spaces during pretraining of EuroLLM.<n>We find that shared concept spaces emerge early and continue to refine, but that alignment with them is language-dependent.<n>In contrast to prior work, our fine-grained manual analysis reveals that some apparent gains in translation quality reflect shifts in behavior.
  arXiv  Detail & Related papers  (2026-01-30T11:23:01Z)
• Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localization [20.179748846776995]
  We propose ActionVLM, a vision-language aggregation framework that mitigates modality bias in TAL.<n>Our key insight is to preserve vision as the dominant signal while adaptively exploiting language only when beneficial.<n>Experiments on THUMOS14 show that our model outperforms state-of-the-art by up to 3.2% mAP.
  arXiv  Detail & Related papers  (2026-01-28T22:03:46Z)
• Stable Language Guidance for Vision-Language-Action Models [62.80963701282789]
  Residual Semantic Steering is a probabilistic framework that disentangles physical affordance from semantic execution.<n> RSS achieves state-of-the-art robustness, maintaining performance even under adversarial linguistic perturbations.
  arXiv  Detail & Related papers  (2026-01-07T16:16:10Z)
• CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention [56.08815340137881]
  Large Vision-Language Models (LVLMs) have demonstrated impressive multimodal abilities but remain prone to multilingual object hallucination.<n>We propose Cross-Lingual Attention Intervention for Mitigating multilingual object hallucination (CLAIM) in LVLMs, a novel near training-free method by aligning attention patterns.
  arXiv  Detail & Related papers  (2025-06-03T11:17:16Z)
• Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination [13.706325901731665]
  Multimodal large language models (MLLMs) have advanced the integration of visual and linguistic modalities. Current approaches like chain of thought (CoT) reasoning have augmented the cognitive capabilities of large language models (LLMs) But their adaptation to MLLMs is hindered by heightened risks of hallucination in cross-modality comprehension.
  arXiv  Detail & Related papers  (2024-11-15T21:01:37Z)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization [52.935150075484074]
  We introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously.
  arXiv  Detail & Related papers  (2023-09-09T03:01:38Z)
• Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting [105.5303416210736]
  Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. It is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising.
  arXiv  Detail & Related papers  (2020-05-06T20:11:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.