Deep Language Geometry: Constructing a Metric Space from LLM Weights

• URL: http://arxiv.org/abs/2508.11676v1
• Date: Fri, 08 Aug 2025 13:48:26 GMT
• Title: Deep Language Geometry: Constructing a Metric Space from LLM Weights
• Authors: Maksym Shamrai, Vladyslav Hamolia,
• Abstract summary: We introduce a novel framework that utilizes the internal weight activations of modern Large Language Models (LLMs) to construct a metric space of languages.<n>Our method automatically derives high-dimensional vector representations by computing weight importance scores via an adapted pruning algorithm.<n>We validate our approach across diverse datasets and multilingual LLMs, covering 106 languages.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a novel framework that utilizes the internal weight activations of modern Large Language Models (LLMs) to construct a metric space of languages. Unlike traditional approaches based on hand-crafted linguistic features, our method automatically derives high-dimensional vector representations by computing weight importance scores via an adapted pruning algorithm. Our approach captures intrinsic language characteristics that reflect linguistic phenomena. We validate our approach across diverse datasets and multilingual LLMs, covering 106 languages. The results align well with established linguistic families while also revealing unexpected inter-language connections that may indicate historical contact or language evolution. The source code, computed language latent vectors, and visualization tool are made publicly available at https://github.com/mshamrai/deep-language-geometry.

Related papers

• Language Steering for Multilingual In-Context Learning [10.932074928744568]
  Large language models' performance on non-English languages remains substantially inferior to English.<n>We propose language vectors -- a training-free language steering approach.<n>We show consistent improvements on multilingual in-context learning over baselines across all tasks and languages tested.
  arXiv  Detail & Related papers  (2026-02-02T16:52:09Z)
• Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders [51.380449540006985]
  Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear.<n>Do they form shared multilingual representations with language-specific decoding, and if so, why does performance still favor the dominant training language?<n>We analyze their internal mechanisms using cross-layer transcoders (CLT) and attribution graphs.
  arXiv  Detail & Related papers  (2025-11-13T22:51:06Z)
• Lens: Rethinking Multilingual Enhancement for Large Language Models [70.85065197789639]
  We propose Lens, a novel approach to enhance multilingual capabilities in large language models (LLMs)<n>Lens operates on two subspaces: the language-agnostic subspace, where it aligns target languages with the central language to inherit strong semantic representations, and the language-specific subspace, where it separates target and central languages to preserve linguistic specificity.<n>Lens significantly improves multilingual performance while maintaining the model's English proficiency, achieving better results with less computational cost compared to existing post-training approaches.
  arXiv  Detail & Related papers  (2024-10-06T08:51:30Z)
• Counterfactually Probing Language Identity in Multilingual Models [15.260518230218414]
  We use AlterRep, a method of counterfactual probing, to explore the internal structure of multilingual models. We find that, given a template in Language X, pushing towards Language Y systematically increases the probability of Language Y words.
  arXiv  Detail & Related papers  (2023-10-29T01:21:36Z)
• Robust Open-Set Spoken Language Identification and the CU MultiLang Dataset [2.048226951354646]
  Open-set spoken language identification systems can detect when an input exhibits none of the original languages. We implement a novel approach to open-set spoken language identification that uses MFCC and pitch features. We present a spoken language identification system that achieves 91.76% accuracy on trained languages and has the capability to adapt to unknown languages on the fly.
  arXiv  Detail & Related papers  (2023-08-29T00:44:27Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• The Geometry of Multilingual Language Models: An Equality Lens [2.6746119935689214]
  We analyze the geometry of three multilingual language models in Euclidean space. Using a geometric separability index we find that although languages tend to be closer according to their linguistic family, they are almost separable with languages from other families.
  arXiv  Detail & Related papers  (2023-05-13T05:19:15Z)
• Revisiting Language Encoding in Learning Multilingual Representations [70.01772581545103]
  We propose a new approach called Cross-lingual Language Projection (XLP) to replace language embedding. XLP projects the word embeddings into language-specific semantic space, and then the projected embeddings will be fed into the Transformer model. Experiments show that XLP can freely and significantly boost the model performance on extensive multilingual benchmark datasets.
  arXiv  Detail & Related papers  (2021-02-16T18:47:10Z)
• Automatically Identifying Language Family from Acoustic Examples in Low Resource Scenarios [48.57072884674938]
  We propose a method to analyze language similarity using deep learning. Namely, we train a model on the Wilderness dataset and investigate how its latent space compares with classical language family findings.
  arXiv  Detail & Related papers  (2020-12-01T22:44:42Z)
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision [110.66085917826648]
  We develop a technique that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to their related images. "vokenization" is trained on relatively small image captioning datasets and we then apply it to generate vokens for large language corpora. Trained with these contextually generated vokens, our visually-supervised language models show consistent improvements over self-supervised alternatives on multiple pure-language tasks.
  arXiv  Detail & Related papers  (2020-10-14T02:11:51Z)
• Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations [83.27475281544868]
  We use singular vector canonical correlation analysis to study what kind of information is induced from each source. We observe that our representations embed typology and strengthen correlations with language relationships. We then take advantage of our multi-view language vector space for multilingual machine translation, where we achieve competitive overall translation accuracy.
  arXiv  Detail & Related papers  (2020-04-30T16:25:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.