When Language Representations Interact: Separability and Cross-Lingual Effects in LLMs

• URL: http://arxiv.org/abs/2606.14347v1
• Date: Fri, 12 Jun 2026 11:00:59 GMT
• Title: When Language Representations Interact: Separability and Cross-Lingual Effects in LLMs
• Abstract summary: Large language models exhibit strong multilingual capabilities, but their internal representations are difficult to interpret.<n>Recent work has shown that causal-geometric structure can explain how certain concepts are encoded as approximately linear and separable directions.<n>We apply causal-geometric analysis to multilingual LLMs, studying 28 bilingual contrasts across three models.
• Score: 20.39151103511549
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models exhibit strong multilingual capabilities, however, their internal representations are difficult to interpret. Understanding these interactions is important for ensuring reliable behavior in multilingual systems. Recent work has shown that causal-geometric structure can explain how certain concepts are encoded as approximately linear and separable directions, but whether this framework extends to multilingual models, where language identity is correlated and hierarchical, is underexplored. We apply causal-geometric analysis to multilingual LLMs, studying 28 bilingual contrasts across three models, allowing us to analyze when languages behave as approximately independent factors and when structured dependencies persist. We find evidence that language concepts admit stable linear representations that are largely separable under a covariance-adjusted (causal) inner product, with structured deviations reflecting linguistic similarity. Moreover, languages within the same family (such as Germanic or Romance) exhibit a simplex-like geometric structure, suggesting hierarchical organization. These results extend causal-geometric interpretability to multilingual settings and provide insight into how separability and similarity may exist in multilingual LLM representations, motivating interpretability analyses that diagnose when and how structured dependencies between concepts can be anticipated. This has implications for trustworthy deployment, as residual structure between languages may lead to unintended cross-lingual effects when models are monitored or intervened upon.

Related papers

• Multilinguality of Large Language Models From a Structural Perspective [55.097821802807566]
  Large language models (LLMs) have excelled in processing multiple languages through pre- and post-training on multilingual data.<n>Our findings reveal that low-resource languages are structurally more different from English than high- and mid-resource languages.
  arXiv  Detail & Related papers  (2026-06-01T07:18:09Z)
• StructLens: A Structural Lens for Language Models via Maximum Spanning Trees [52.040177523973334]
  StructLens is an analytical framework designed to reveal how internal structures relate holistically.<n>Our findings demonstrate that StructLens yields an inter-layer similarity pattern that is distinctively different from conventional cosine similarity.
  arXiv  Detail & Related papers  (2026-02-10T11:30:32Z)
• On the Entity-Level Alignment in Crosslingual Consistency [62.33186691736433]
  SubSub and SubInj integrate English translations of subjects into prompts across languages, leading to substantial gains in factual recall accuracy and consistency.<n>These interventions reinforce the entity representation alignment in the conceptual space through model's internal pivot-language processing.
  arXiv  Detail & Related papers  (2025-10-11T16:26:50Z)
• Neighbors and relatives: How do speech embeddings reflect linguistic connections across the world? [0.7168794329741259]
  This study employs embeddings from the fine-tuned XLS-R self-supervised language identification model vox107-xls-r-300m-wav2vec to analyze relationships between 106 world languages.<n>Using linear discriminant analysis (LDA), language embeddings are clustered and compared with genealogical, lexical, and geographical distances.<n>The results demonstrate that embedding-based distances align closely with traditional measures, effectively capturing both global and local typological patterns.
  arXiv  Detail & Related papers  (2025-06-10T08:33:34Z)
• Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models [56.61984030508691]
  We present the first mechanistic interpretability study of language confusion.<n>We show that confusion points (CPs) are central to this phenomenon.<n>We show that editing a small set of critical neurons, identified via comparative analysis with a multilingual-tuned counterpart, substantially mitigates confusion.
  arXiv  Detail & Related papers  (2025-05-22T11:29:17Z)
• When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners [111.50503126693444]
  We show that language-specific ablation consistently boosts multilingual reasoning performance.<n>Compared to post-training, our training-free ablation achieves comparable or superior results with minimal computational overhead.
  arXiv  Detail & Related papers  (2025-05-21T08:35:05Z)
• High-Dimensional Interlingual Representations of Large Language Models [65.77317753001954]
  Large language models (LLMs) trained on massive multilingual datasets hint at the formation of interlingual constructs.<n>We explore 31 diverse languages varying on their resource-levels, typologies, and geographical regions.<n>We find that multilingual LLMs exhibit inconsistent cross-lingual alignments.
  arXiv  Detail & Related papers  (2025-03-14T10:39:27Z)
• Constructive Approach to Bidirectional Influence between Qualia Structure and Language Emergence [5.906966694759679]
  This perspective paper explores the bidirectional influence between language emergence and the structure of subjective experiences.<n>We hypothesize that the emergence of languages with distributional semantics is linked to the coordination of internal representations shaped by experience.
  arXiv  Detail & Related papers  (2024-09-14T11:03:12Z)
• A Joint Matrix Factorization Analysis of Multilingual Representations [28.751144371901958]
  We present an analysis tool based on joint matrix factorization for comparing latent representations of multilingual and monolingual models. We study to what extent and how morphosyntactic features are reflected in the representations learned by multilingual pre-trained models.
  arXiv  Detail & Related papers  (2023-10-24T04:43:45Z)
• Cross-Lingual Transfer of Cognitive Processing Complexity [11.939409227407769]
  We use sentence-level eye-tracking patterns as a cognitive indicator for structural complexity. We show that the multilingual model XLM-RoBERTa can successfully predict varied patterns for 13 typologically diverse languages.
  arXiv  Detail & Related papers  (2023-02-24T15:48:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.