One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representations

• URL: http://arxiv.org/abs/2603.08869v1
• Date: Mon, 09 Mar 2026 19:31:20 GMT
• Title: One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representations
• Abstract summary: We investigate whether features learned by Sparse Autoencoders (SAEs) represent abstract meaning, or are tied to how text is written.<n>We find that identical sentences in different Serbian scripts activate highly overlapping features, far exceeding random baselines.<n>Our findings suggest SAE features can capture semantics at a level of abstraction above surface tokenization.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Do the features learned by Sparse Autoencoders (SAEs) represent abstract meaning, or are they tied to how text is written? We investigate this question using Serbian digraphia as a controlled testbed: Serbian is written interchangeably in Latin and Cyrillic scripts with a near-perfect character mapping between them, enabling us to vary orthography while holding meaning exactly constant. Crucially, these scripts are tokenized completely differently, sharing no tokens whatsoever. Analyzing SAE feature activations across the Gemma model family (270M-27B parameters), we find that identical sentences in different Serbian scripts activate highly overlapping features, far exceeding random baselines. Strikingly, changing script causes less representational divergence than paraphrasing within the same script, suggesting SAE features prioritize meaning over orthographic form. Cross-script cross-paraphrase comparisons provide evidence against memorization, as these combinations rarely co-occur in training data yet still exhibit substantial feature overlap. This script invariance strengthens with model scale. Taken together, our findings suggest that SAE features can capture semantics at a level of abstraction above surface tokenization, and we propose Serbian digraphia as a general evaluation paradigm for probing the abstractness of learned representations.

Related papers

• How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings [0.0]
  We show that SAE feature sets activated by the same content in different languages, scripts, and wordings share substantial overlap.<n>We then test whether auto-interpretation labels keep pace. They often do not.<n>These results suggest that auto-interpretation labels may reflect a feature's behavior on well-represented inputs rather than the concept itself.
  arXiv  Detail & Related papers  (2026-05-29T20:59:28Z)
• The Latin Substrate: How Language Models Represent and Mediate Script Choice [10.624509426092597]
  We show that scripts of the same language become increasingly separable across layers.<n>We localize a small set of late-layer attention heads that causally mediate script choice.
  arXiv  Detail & Related papers  (2026-05-29T14:36:04Z)
• ScriptHOI: Learning Scripted State Transitions for Open-Vocabulary Human-Object Interaction Detection [1.2710597393824294]
  Open-vocabulary human-object interaction (HOI) detection requires recognizing interaction phrases that may not appear as categories during training.<n>Recent vision-language HOI detectors improve semantic transfer by matching human-object features with text embeddings.<n>We propose ScriptHOI, a structured framework that represents each interaction phrase as a soft scripted state transition.
  arXiv  Detail & Related papers  (2026-05-06T15:52:35Z)
• Empirical Evaluation of Progressive Coding for Sparse Autoencoders [45.94517951918044]
  We show that dictionary importance in vanilla SAEs follows a power law.<n>We show Matryoshka SAEs exhibit lower reconstruction loss and recaptured language modeling loss.
  arXiv  Detail & Related papers  (2025-04-30T21:08:32Z)
• Unified Lexical Representation for Interpretable Visual-Language Alignment [52.059812317944434]
  We introduce LexVLA, a framework for learning a unified lexical representation for both modalities without complex design. We use DINOv2 as our visual model for its local-inclined features and Llama 2, a generative language model, to leverage its in-context lexical prediction ability. We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning on the modest multi-modal dataset.
  arXiv  Detail & Related papers  (2024-07-25T07:35:27Z)
• Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scripts [50.40191599304911]
  We investigate whether transliteration is also effective in improving LLMs' performance for low-resource languages written in non-Latin scripts. We propose three prompt templates, where the target-language text is represented in (1) its original script, (2) Latin script, or (3) both. Our findings show that the effectiveness of transliteration varies by task type and model size.
  arXiv  Detail & Related papers  (2024-07-02T14:51:20Z)
• Lexinvariant Language Models [84.2829117441298]
  Token embeddings, a mapping from discrete lexical symbols to continuous vectors, are at the heart of any language model (LM) We study textitlexinvariantlanguage models that are invariant to lexical symbols and therefore do not need fixed token embeddings in practice. We show that a lexinvariant LM can attain perplexity comparable to that of a standard language model, given a sufficiently long context.
  arXiv  Detail & Related papers  (2023-05-24T19:10:46Z)
• STAIR: Learning Sparse Text and Image Representation in Grounded Tokens [84.14528645941128]
  We show that it is possible to build a sparse semantic representation that is as powerful as, or even better than, dense presentations. We extend the CLIP model and build a sparse text and image representation (STAIR), where the image and text are mapped to a sparse token space. It significantly outperforms a CLIP model with +$4.9%$ and +$4.3%$ absolute Recall@1 improvement.
  arXiv  Detail & Related papers  (2023-01-30T17:21:30Z)
• What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary [68.77983831618685]
  We propose to interpret the vector representations produced by dual encoders by projecting them into the model's vocabulary space. We show that the resulting projections contain rich semantic information, and draw connection between them and sparse retrieval.
  arXiv  Detail & Related papers  (2022-12-20T16:03:25Z)
• Exposing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders [85.80950708769923]
  We probe multilingual language models for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a novel method to expose this knowledge by additionally fine-tuning multilingual models. We report substantial gains on standard benchmarks.
  arXiv  Detail & Related papers  (2022-04-30T13:23:16Z)
• Does Transliteration Help Multilingual Language Modeling? [0.0]
  We empirically measure the effect of transliteration on Multilingual Language Models. We focus on the Indic languages, which have the highest script diversity in the world. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high-resource languages.
  arXiv  Detail & Related papers  (2022-01-29T05:48:42Z)
• Disentangling Homophemes in Lip Reading using Perplexity Analysis [10.262299768603894]
  This paper proposes a new application for the Generative Pre-Training transformer. It serves as a language model to convert visual speech in the form of visemes, to language in the form of words and sentences. The network uses the search for optimal perplexity to perform the viseme-to-word mapping.
  arXiv  Detail & Related papers  (2020-11-28T12:12:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.