The Latin Substrate: How Language Models Represent and Mediate Script Choice

• URL: http://arxiv.org/abs/2605.31363v1
• Date: Fri, 29 May 2026 14:36:04 GMT
• Title: The Latin Substrate: How Language Models Represent and Mediate Script Choice
• Abstract summary: We show that scripts of the same language become increasingly separable across layers.<n>We localize a small set of late-layer attention heads that causally mediate script choice.
• Score: 10.624509426092597
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many languages are written in multiple scripts, requiring large language models (LLMs) to generate equivalent linguistic content in distinct orthographic forms. While prior work suggests that LLMs route information through shared latent representations, how they internally mediate script variation remains poorly understood. We study this question by first examining per-layer output distributions with the logit lens, which reveals consistent latent romanization during transliteration, and then through representational and mechanistic analyses of script generation. At the representational level, we show that scripts of the same language become increasingly separable across layers and that a simple linear steering direction can flip a model's output script while largely maintaining semantic content. The vector generalizes asymmetrically to writing systems unseen during construction, flipping non-Latin output to Latin reliably, but mapping Latin output into varied non-Latin scripts. At the mechanistic level, we localize a small set of late-layer attention heads that causally mediate script choice. These heads transfer across unrelated languages and writing systems, suggesting that script routing is implemented by language-agnostic components. Across both analyses, we observe a consistent directional asymmetry: non-Latin output is produced by a compact, identifiable gate, while Latin-script output emerges from diffuse contributions across the network. Collectively, our findings hint that LLMs organize script variation around shared latent representations while exhibiting a privileged substrate toward Latin script.

Related papers

• One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representations [0.0]
  We investigate whether features learned by Sparse Autoencoders (SAEs) represent abstract meaning, or are tied to how text is written.<n>We find that identical sentences in different Serbian scripts activate highly overlapping features, far exceeding random baselines.<n>Our findings suggest SAE features can capture semantics at a level of abstraction above surface tokenization.
  arXiv  Detail & Related papers  (2026-03-09T19:31:20Z)
• Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration [70.84108518476744]
  We show that script is linearly encoded in the activation space of multilingual speech models, and that modifying activations at inference time enables direct control over output script.<n>We apply this approach to inducing post-hoc control over the script of speech recognition output, where we observe competitive performance across all model sizes of Whisper.
  arXiv  Detail & Related papers  (2026-01-06T10:45:04Z)
• Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders [51.380449540006985]
  Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear.<n>Do they form shared multilingual representations with language-specific decoding, and if so, why does performance still favor the dominant training language?<n>We analyze their internal mechanisms using cross-layer transcoders (CLT) and attribution graphs.
  arXiv  Detail & Related papers  (2025-11-13T22:51:06Z)
• RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs [18.27925188037189]
  Large Language Models (LLMs) exhibit strong multilingual performance despite being predominantly trained on English-centric corpora.<n>This raises a fundamental question: How do LLMs achieve such multilingual capabilities?<n> Focusing on languages written in non-Roman scripts, we investigate the role of Romanization as a potential bridge in multilingual processing.
  arXiv  Detail & Related papers  (2025-02-11T10:10:26Z)
• Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languages [37.61699757912346]
  We propose leveraging phonemic transcriptions as complementary signals to induce script-invariant representations.<n>We show that phonemic and orthographic scripts retrieve distinct examples for in-context learning (ICL)<n>This motivates our proposed Mixed-ICL retrieval strategy, where further aggregation from both leads to significant performance improvements.
  arXiv  Detail & Related papers  (2024-11-04T18:59:51Z)
• Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scripts [50.40191599304911]
  We investigate whether transliteration is also effective in improving LLMs' performance for low-resource languages written in non-Latin scripts. We propose three prompt templates, where the target-language text is represented in (1) its original script, (2) Latin script, or (3) both. Our findings show that the effectiveness of transliteration varies by task type and model size.
  arXiv  Detail & Related papers  (2024-07-02T14:51:20Z)
• Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment [50.27950279695363]
  The transfer performance is often hindered when a low-resource target language is written in a different script than the high-resource source language. Inspired by recent work that uses transliteration to address this problem, our paper proposes a transliteration-based post-pretraining alignment (PPA) method.
  arXiv  Detail & Related papers  (2024-06-28T08:59:24Z)
• MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling [70.34758460372629]
  We introduce a new paradigm that encodes the same information with segments of consistent size across diverse languages. MYTE produces shorter encodings for all 99 analyzed languages. This, in turn, improves multilingual LM performance and diminishes the perplexity gap throughout diverse languages.
  arXiv  Detail & Related papers  (2024-03-15T21:21:11Z)
• TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models [50.40191599304911]
  We propose TransliCo to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script. We show that Furina outperforms the original Glot500-m on various zero-shot crosslingual transfer tasks.
  arXiv  Detail & Related papers  (2024-01-12T15:12:48Z)
• Does Transliteration Help Multilingual Language Modeling? [0.0]
  We empirically measure the effect of transliteration on Multilingual Language Models. We focus on the Indic languages, which have the highest script diversity in the world. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high-resource languages.
  arXiv  Detail & Related papers  (2022-01-29T05:48:42Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.