A Glyph Is Not a Letter, a Token Is Not a Word, a Space Is Not a Space: What the Units of Voynichese Are Not

• URL: http://arxiv.org/abs/2608.17096v1
• Date: Mon, 17 Aug 2026 20:05:12 GMT
• Title: A Glyph Is Not a Letter, a Token Is Not a Word, a Space Is Not a Space: What the Units of Voynichese Are Not
• Abstract summary: We analyse the Voynich manuscript (Beinecke MS 408) with matched prose, cipher, and pseudo-text controls and quire-level resampling.<n>None holds, and the failures share a shape: the order in Voynichese sits at the edges of tokens and at graded boundaries between them.<n>A published Voynich-imitating cipher and a self-citation text generator both reproduce the low entropy, the unit scale, the weak token order, and the null result of a calibrated substitution attack.
• Score: 45.88028371034407
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Voynich manuscript (Beinecke MS 408) is usually analysed on three unstated assumptions: that its glyphs are letters, that the strings between blanks are words, and that every blank is a word space. We test all three against the Zandbergen-Landini transliteration with matched prose, cipher, and pseudo-text controls and quire-level resampling. None holds, and the failures share a shape: the order in Voynichese sits at the edges of tokens and at graded boundaries between them, not in the succession of tokens themselves. Glyph regularity is too strong for one-to-one substitution of any tested plaintext (conditional entropy 2.7 bits against about 3.5 for Latin, Italian, and English) and resolves instead onto a quire-stable scale of recurrent multi-symbol units. Tokens form a plausible vocabulary, yet the identity of one token predicts the next by under 1% of token entropy, below every matched control (2-10%), while the glyphs at token edges share 0.2 bits of mutual information, more than in any prose control. Blanks fall into two regimes: the separators transcribers marked uncertain behave like word-internal junctures, are physically narrower on the page (AUC 0.905 from independent image coordinates, with the same sign in a small blind ink audit), and are crossed by learned units even when every space is erased before learning. This profile is also what discriminates. A published Voynich-imitating cipher and a self-citation text generator both reproduce the low entropy, the unit scale, the weak token order, and the null result of a calibrated substitution attack; neither reproduces the edge-glyph coupling or the open, hapax-rich vocabulary (70% singleton types against 41% and 59-60%). Any account of the manuscript must therefore earn, rather than assume, the step from glyphs, tokens, and separators to letters, words, and word spaces, and these are the measurements on which to do so.

Related papers

• Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses [0.0]
  We use a training-free tracer that reads at every position, sub-layer, and head average.<n>It traces a compound- noun->attention chain in Phi-2, confirmed there by activation patching.<n>It reads what retrieval surfaces for real versus fictional entities, and reads metaphor as a set of domain-to-domain mappings.
  arXiv  Detail & Related papers  (2026-09-09T09:01:09Z)
• Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility [0.0]
  In abugida scripts, vowels are written as combining marks; this pattern therefore splits each word at every vowel sign.<n>Since BPE merges only within a pre-token, those splits persist through training regardless of vocabulary size or corpus composition.<n>Across 26 languages from a parallel corpus, every one of the 17 abugidas is affected, ranging from 1.47x (Tibetan) to 9.02x (Thai)<n>For 5 languages, matched tokenizer pairs that differ only in this character class fall within 2.2% of the predicted floor, scoring 4.78 versus 1.58 tokens per word on Nepali.
  arXiv  Detail & Related papers  (2026-08-26T22:56:24Z)
• Explicit Boundary Markers for Subword Vocabularies [12.858857146500414]
  Subword tokenizers represent many common words twice in space-using writing systems.<n>We introduce an alternative to standard whitespace conventions using an explicit word boundary marker.
  arXiv  Detail & Related papers  (2026-08-09T18:16:51Z)
• The Learning Objective Governs Perceptual Narrowing: A Cross-Lingual, Layer-Wise, Ten-Seed Study of Self-Supervised Speech Encoders [0.0]
  We train a Transformer encoder on child-directed and read speech and evaluate phoneme ABX in English, French, and Mandarin over ten seeds.<n>We conclude that the objective, not the architecture, is the first-order determinant of narrowing-shaped representational change.
  arXiv  Detail & Related papers  (2026-08-01T08:02:35Z)
• Metaphor Tracer: A Theory-Informed Analysis of Hidden States [0.0]
  From one forward pass, without training, we score every token position on two properties.<n>The *aggregator* measures whether the position consolidates the whole text into a stable configuration.<n>The *differentiator*, whether other tokens are transiently carried into its subspace as the model reads.
  arXiv  Detail & Related papers  (2026-07-30T16:12:13Z)
• Are you going to finish that? A Practical Study of the Partial Token Problem [85.49816027251013]
  Language models (LMs) are trained over sequences of tokens, whereas users interact with LMs via text.<n>This mismatch gives rise to the partial token problem, which occurs when a user ends their prompt in the middle of the expected next-token.<n>In this work, we identify three domains where token and "word" boundaries often do not line up.
  arXiv  Detail & Related papers  (2026-01-30T17:47:16Z)
• Levée d'ambiguïtés par grammaires locales [0.0]
  This article concerns a lexical disambiguation method adapted to the objective of a zero silence rate and implemented in Silberztein's INTEX system (1993).<n>We show that to verify a local disambiguation grammar in this framework, it is not sufficient to consider the transducer paths separately.
  arXiv  Detail & Related papers  (2025-10-28T15:38:22Z)
• TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection [62.95726973851089]
  TokenCLIP is a token-wise adaptation framework for anomaly learning.<n>It enables dynamic alignment between visual and learnable textual spaces for fine-grained anomaly learning.
  arXiv  Detail & Related papers  (2025-10-24T05:51:31Z)
• Bukva: Russian Sign Language Alphabet [75.42794328290088]
  This paper investigates the recognition of the Russian fingerspelling alphabet, also known as the Russian Sign Language (RSL) dactyl. Dactyl is a component of sign languages where distinct hand movements represent individual letters of a written language. We provide Bukva, the first full-fledged open-source video dataset for RSL dactyl recognition.
  arXiv  Detail & Related papers  (2024-10-11T09:59:48Z)
• Provably Secure Disambiguating Neural Linguistic Steganography [66.30965740387047]
  The segmentation ambiguity problem, which arises when using language models based on subwords, leads to occasional decoding failures.<n>We propose a novel secure disambiguation method named SyncPool, which effectively addresses the segmentation ambiguity problem.<n> SyncPool does not change the size of the candidate pool or the distribution of tokens and thus is applicable to provably secure language steganography methods.
  arXiv  Detail & Related papers  (2024-03-26T09:25:57Z)
• Lexinvariant Language Models [84.2829117441298]
  Token embeddings, a mapping from discrete lexical symbols to continuous vectors, are at the heart of any language model (LM) We study textitlexinvariantlanguage models that are invariant to lexical symbols and therefore do not need fixed token embeddings in practice. We show that a lexinvariant LM can attain perplexity comparable to that of a standard language model, given a sufficiently long context.
  arXiv  Detail & Related papers  (2023-05-24T19:10:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.