Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures

• URL: http://arxiv.org/abs/2607.22699v1
• Date: Sat, 18 Jul 2026 11:49:20 GMT
• Title: Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures
• Abstract summary: Large Language Models (LLMs) are known to struggle generalizing multilingually, to languages outside of English.<n>We show how well LLMs' representations capture the hierarchical similarity structure between distinct languages.<n>This extends classic work on similarity-driven generalization at scale, showing how models that represent similar languages similarly generalize better from one language to another.
• Score: 3.1931511014508533
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Large Language Models (LLMs) grow more capable across diverse tasks, their (in)ability to generalize remains difficult to quantify and poorly understood beyond limited domains. In particular, LLMs are known to struggle generalizing multilingually, to languages outside of English, and that are poorly attested in their training data. To understand why this may be, and what enables some models to perform better than others, we turn to a long history of work across the cognitive sciences, arguing that successful generalization derives from appropriate representations in similarity space. We look at how well LLMs' representations capture the hierarchical similarity structure between distinct languages. Strikingly, we show LLMs' latent representations largely recover the hierarchical structure of the Indo-European language family tree -- grouping languages that are members of the same subfamily closely together in representation space. Furthermore, we show that the degree to which models reflect the similarity structure of languages correlates with their performance on XNLI, a multilingual natural language inference benchmark. This extends classic work on similarity-driven generalization at scale, showing how models that represent similar languages similarly generalize better from one language to another.

Related papers

• When Language Representations Interact: Separability and Cross-Lingual Effects in LLMs [20.39151103511549]
  Large language models exhibit strong multilingual capabilities, but their internal representations are difficult to interpret.<n>Recent work has shown that causal-geometric structure can explain how certain concepts are encoded as approximately linear and separable directions.<n>We apply causal-geometric analysis to multilingual LLMs, studying 28 bilingual contrasts across three models.
  arXiv  Detail & Related papers  (2026-06-12T11:00:59Z)
• Multilinguality of Large Language Models From a Structural Perspective [55.097821802807566]
  Large language models (LLMs) have excelled in processing multiple languages through pre- and post-training on multilingual data.<n>Our findings reveal that low-resource languages are structurally more different from English than high- and mid-resource languages.
  arXiv  Detail & Related papers  (2026-06-01T07:18:09Z)
• Shaping Schema via Language Representation as the Next Frontier for LLM Intelligence Expanding [88.11781604392606]
  This paper defines language representation as the linguistic and symbolic constructs used to map and model the real world.<n>We argue that shaping schemas through advanced language representation is the next frontier for expanding Large Language Models.
  arXiv  Detail & Related papers  (2026-05-10T02:42:29Z)
• When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners [111.50503126693444]
  We show that language-specific ablation consistently boosts multilingual reasoning performance.<n>Compared to post-training, our training-free ablation achieves comparable or superior results with minimal computational overhead.
  arXiv  Detail & Related papers  (2025-05-21T08:35:05Z)
• Language-Specific Latent Process Hinders Cross-Lingual Performance [38.36668133949413]
  Large language models (LLMs) are capable of cross-lingual transfer, but can produce inconsistent output when prompted with the same queries written in different languages.<n>We measure representation similarity between languages, and apply the logit lens to interpret the implicit steps taken by LLMs to solve multilingual multi-choice reasoning questions.<n>Our analyses reveal LLMs predict inconsistently and are less accurate because they rely on representations that are dissimilar across languages, rather than working in a shared semantic space.
  arXiv  Detail & Related papers  (2025-05-19T14:10:15Z)
• High-Dimensional Interlingual Representations of Large Language Models [65.77317753001954]
  Large language models (LLMs) trained on massive multilingual datasets hint at the formation of interlingual constructs.<n>We explore 31 diverse languages varying on their resource-levels, typologies, and geographical regions.<n>We find that multilingual LLMs exhibit inconsistent cross-lingual alignments.
  arXiv  Detail & Related papers  (2025-03-14T10:39:27Z)
• How Proficient Are Large Language Models in Formal Languages? An In-Depth Insight for Knowledge Base Question Answering [52.86931192259096]
  Knowledge Base Question Answering (KBQA) aims to answer natural language questions based on facts in knowledge bases. Recent works leverage the capabilities of large language models (LLMs) for logical form generation to improve performance.
  arXiv  Detail & Related papers  (2024-01-11T09:27:50Z)
• Probing LLMs for Joint Encoding of Linguistic Categories [10.988109020181563]
  We propose a framework for testing the joint encoding of linguistic categories in Large Language Models (LLMs) We find evidence of joint encoding both at the same (related part-of-speech (POS) classes) and different (POS classes and related syntactic dependency relations) levels of linguistic hierarchy.
  arXiv  Detail & Related papers  (2023-10-28T12:46:40Z)
• Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization [27.368684663279463]
  We investigate the potential for explicitly aligning conceptual correspondence between languages to enhance cross-lingual generalization. Using the syntactic aspect of language as a testbed, our analyses of 43 languages reveal a high degree of alignability. We propose a meta-learning-based method to learn to align conceptual spaces of different languages.
  arXiv  Detail & Related papers  (2023-10-19T14:50:51Z)
• GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding [74.39024160277809]
  We present Global--Local Contrastive Learning Framework (GL-CLeF) to address this shortcoming. Specifically, we employ contrastive learning, leveraging bilingual dictionaries to construct multilingual views of the same utterance. GL-CLeF achieves the best performance and successfully pulls representations of similar sentences across languages closer.
  arXiv  Detail & Related papers  (2022-04-18T13:56:58Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.