Language Lives in Sparse Dimensions: Toward Interpretable and Efficient Multilingual Control for Large Language Models

• URL: http://arxiv.org/abs/2510.07213v1
• Date: Wed, 08 Oct 2025 16:46:57 GMT
• Title: Language Lives in Sparse Dimensions: Toward Interpretable and Efficient Multilingual Control for Large Language Models
• Abstract summary: Large language models exhibit strong multilingual capabilities despite limited exposure to non-English data.<n>We show that English-centric large language models map multilingual content into English-aligned representations at intermediate layers and then project them back into target-language token spaces in the final layer.<n>We introduce a simple, training-free method to identify and manipulate these dimensions, requiring only as few as 50 sentences of either parallel or monolingual data.
• Score: 39.550695697979855
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models exhibit strong multilingual capabilities despite limited exposure to non-English data. Prior studies show that English-centric large language models map multilingual content into English-aligned representations at intermediate layers and then project them back into target-language token spaces in the final layer. From this observation, we hypothesize that this cross-lingual transition is governed by a small and sparse set of dimensions, which occur at consistent indices across the intermediate to final layers. Building on this insight, we introduce a simple, training-free method to identify and manipulate these dimensions, requiring only as few as 50 sentences of either parallel or monolingual data. Experiments on a multilingual generation control task reveal the interpretability of these dimensions, demonstrating that the interventions in these dimensions can switch the output language while preserving semantic content, and that it surpasses the performance of prior neuron-based approaches at a substantially lower cost.

Related papers

• English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training [7.141617235075284]
  Post-training pipelines remain predominantly English-centric, contributing to performance disparities across languages.<n>We present a systematic, controlled study of the interplay between training language coverage, model scale, and task domain.<n>We find that increasing language coverage during post-training is largely beneficial across tasks and model scales.
  arXiv  Detail & Related papers  (2026-04-14T20:26:34Z)
• Language Steering for Multilingual In-Context Learning [10.932074928744568]
  Large language models' performance on non-English languages remains substantially inferior to English.<n>We propose language vectors -- a training-free language steering approach.<n>We show consistent improvements on multilingual in-context learning over baselines across all tasks and languages tested.
  arXiv  Detail & Related papers  (2026-02-02T16:52:09Z)
• When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training [57.230355403478995]
  We investigate the development of language-agnostic concept spaces during pretraining of EuroLLM.<n>We find that shared concept spaces emerge early and continue to refine, but that alignment with them is language-dependent.<n>In contrast to prior work, our fine-grained manual analysis reveals that some apparent gains in translation quality reflect shifts in behavior.
  arXiv  Detail & Related papers  (2026-01-30T11:23:01Z)
• High-Dimensional Interlingual Representations of Large Language Models [65.77317753001954]
  Large language models (LLMs) trained on massive multilingual datasets hint at the formation of interlingual constructs.<n>We explore 31 diverse languages varying on their resource-levels, typologies, and geographical regions.<n>We find that multilingual LLMs exhibit inconsistent cross-lingual alignments.
  arXiv  Detail & Related papers  (2025-03-14T10:39:27Z)
• Lens: Rethinking Multilingual Enhancement for Large Language Models [70.85065197789639]
  We propose Lens, a novel approach to enhance multilingual capabilities in large language models (LLMs)<n>Lens operates on two subspaces: the language-agnostic subspace, where it aligns target languages with the central language to inherit strong semantic representations, and the language-specific subspace, where it separates target and central languages to preserve linguistic specificity.<n>Lens significantly improves multilingual performance while maintaining the model's English proficiency, achieving better results with less computational cost compared to existing post-training approaches.
  arXiv  Detail & Related papers  (2024-10-06T08:51:30Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora.<n>But can these models relate corresponding concepts across languages, i.e., be crosslingual?<n>This study evaluates state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment [42.624862172666624]
  We propose a simple yet effective cross-lingual alignment framework exploiting pairs of translation sentences. It aligns the internal sentence representations across different languages via multilingual contrastive learning. Experimental results show that even with less than 0.1 textperthousand of pre-training tokens, our alignment framework significantly boosts the cross-lingual abilities of generative language models.
  arXiv  Detail & Related papers  (2023-11-14T11:24:08Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Bootstrapping Multilingual Semantic Parsers using Large Language Models [28.257114724384806]
  translate-train paradigm of transferring English datasets across multiple languages remains to be the key ingredient for training task-specific multilingual models. We consider the task of multilingual semantic parsing and demonstrate the effectiveness and flexibility offered by large language models (LLMs) for translating English datasets into several languages via few-shot prompting.
  arXiv  Detail & Related papers  (2022-10-13T19:34:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.