Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models

• URL: http://arxiv.org/abs/2508.18381v1
• Date: Mon, 25 Aug 2025 18:15:25 GMT
• Title: Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models
• Authors: Yuchun Fan, Yilin Wang, Yongyu Mu, Lei Huang, Bei Li, Xiaocheng Feng, Tong Xiao, Jingbo Zhu,
• Abstract summary: Large vision-language models (LVLMs) have demonstrated exceptional capabilities in understanding visual information with human languages.<n>In this work, we identify a salient correlation between the multilingual understanding ability of LVLMs and language-specific neuron activations in shallow layers.<n>We introduce PLAST, a training recipe that achieves efficient multilingual enhancement for LVLMs by Precise LAnguage-Specific layers fine-Tuning.
• Score: 60.39744129890118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large vision-language models (LVLMs) have demonstrated exceptional capabilities in understanding visual information with human languages but also exhibit an imbalance in multilingual capabilities. In this work, we delve into the multilingual working pattern of LVLMs and identify a salient correlation between the multilingual understanding ability of LVLMs and language-specific neuron activations in shallow layers. Building on this insight, we introduce PLAST, a training recipe that achieves efficient multilingual enhancement for LVLMs by Precise LAnguage-Specific layers fine-Tuning. PLAST first identifies layers involved in multilingual understanding by monitoring language-specific neuron activations. These layers are then precisely fine-tuned with question-translation pairs to achieve multilingual alignment. Our empirical results on MM-Bench and MMMB demonstrate that PLAST effectively improves the multilingual capabilities of LVLMs and achieves significant efficiency with only 14% of the parameters tuned. Further analysis reveals that PLAST can be generalized to low-resource and complex visual reasoning tasks, facilitating the language-specific visual information engagement in shallow layers.

Related papers

• Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models [8.746854869825318]
  We study the contribution of multi-head self-attention in supporting multilingual processing in large language models (LLMs)<n>Applying LAHIS to Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, we reveal the existence of both language-specific and language-general heads.<n>We also introduce a lightweight adaptation that learns a soft head mask to modulate attention outputs over language heads, requiring only 20 tunable parameters to improve XQuAD accuracy.
  arXiv  Detail & Related papers  (2025-11-10T15:12:42Z)
• The Emergence of Abstract Thought in Large Language Models Beyond Any Language [95.50197866832772]
  Large language models (LLMs) function effectively across a diverse range of languages.<n>Preliminary studies observe that the hidden activations of LLMs often resemble English, even when responding to non-English prompts.<n>Recent results show strong multilingual performance, even surpassing English performance on specific tasks in other languages.
  arXiv  Detail & Related papers  (2025-06-11T16:00:54Z)
• The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model [59.357993924917]
  We study the evolution of multilingual capabilities in large language models (LLMs) during the pre-training process.<n>We propose the Babel Tower Hypothesis, which describes the entire process of LLMs acquiring new language capabilities.<n>We propose a novel method to construct an optimized pre-training corpus for multilingual code LLMs.
  arXiv  Detail & Related papers  (2024-12-10T08:28:57Z)
• P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs [84.24644520272835]
  We introduce P-MMEval, a large-scale benchmark covering effective fundamental and capability-specialized datasets.<n>P-MMEval delivers consistent language coverage across various datasets and provides parallel samples.<n>We conduct extensive experiments on representative multilingual model series to compare performances across models and tasks.
  arXiv  Detail & Related papers  (2024-11-14T01:29:36Z)
• Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters [3.7273829129985305]
  This paper explores integration of graph knowledge from linguistic into multilingual Large Models (LLMs)<n>We employ language-specific adapters to improve performance for low-resource languages (LRLs) in sentiment analysis (SA) and named entity recognition (NER)<n>We assess how structured graph knowledge affects the performance of multilingual LLMs for LRLs in SA and NER.
  arXiv  Detail & Related papers  (2024-07-01T15:56:24Z)
• Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners [67.85635044939836]
  Large Language Models (LLMs) have shown impressive language capabilities. In this work, we investigate the spontaneous multilingual alignment improvement of LLMs. We find that LLMs instruction-tuned on the question translation data (i.e. without annotated answers) are able to encourage the alignment between English and a wide range of languages.
  arXiv  Detail & Related papers  (2024-05-22T16:46:19Z)
• How do Large Language Models Handle Multilingualism? [81.15060972112563]
  This study explores how large language models (LLMs) handle multilingualism. LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for thinking and incorporate multilingual knowledge with self-attention and feed-forward structures.
  arXiv  Detail & Related papers  (2024-02-29T02:55:26Z)
• Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models [117.20416338476856]
  Large language models (LLMs) demonstrate remarkable multilingual capabilities without being pre-trained on specially curated multilingual parallel corpora. We propose a novel detection method, language activation probability entropy (LAPE), to identify language-specific neurons within LLMs. Our findings indicate that LLMs' proficiency in processing a particular language is predominantly due to a small subset of neurons.
  arXiv  Detail & Related papers  (2024-02-26T09:36:05Z)
• Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications [24.18102112644796]
  We study the internal neuron activation patterns of large language models (LLMs) when processing different languages. We leverage the discovered differences in expert activation frequencies to guide sparse activation and pruning. Our findings offer new perspectives for applications such as sparse activation and model pruning.
  arXiv  Detail & Related papers  (2024-02-26T07:44:56Z)
• How Vocabulary Sharing Facilitates Multilingualism in LLaMA? [19.136382859468693]
  Large Language Models (LLMs) often show strong performance on English tasks, while exhibiting limitations on other languages. This study endeavors to examine the multilingual capability of LLMs from the vocabulary sharing perspective.
  arXiv  Detail & Related papers  (2023-11-15T16:13:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.