Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

• URL: http://arxiv.org/abs/2606.18717v1
• Date: Wed, 17 Jun 2026 05:55:50 GMT
• Title: Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
• Abstract summary: This paper presents textbfMorpheus, a neural morpheme-boundary model for Turkish.<n>A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships.<n>Morpheus attains the lowest bits-per-character ($1.425$), roughly doubles the gold morphological alignment of the subword family.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text. This paper presents \textbf{Morpheus}, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so $\mathrm{decode}(\mathrm{encode}(w)) = w$ holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word embedding. Among reversible tokenizers -- the only ones valid for generation -- Morpheus attains the lowest bits-per-character ($1.425$), roughly doubles the gold morphological alignment of the subword family (MorphScore macro-F1 $0.61$ vs.\ ${\sim}0.32$), and uses ${\sim}19\%$ less GPU memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical retrieval (root-family MAP $0.85$) and same-root verification (ROC-AUC $1.00$), surpassing the multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead -- a trade-off we attribute to Morpheus's root-centric geometry. Code: https://github.com/lonewolf-rd/TurkishMorpheus; model: https://huggingface.co/lonewolflab/Morpheus-TR-50K; interactive demo: https://huggingface.co/spaces/lonewolflab/morpheus-tr-demo.

Related papers

• MoVoC: Morphology-Aware Subword Construction for Geez Script Languages [7.7761618950496265]
  Subword-based tokenization methods often fail to preserve morphological boundaries.<n>We present MoVoC (Morpheme-aware Subword Vocabulary Construction) and train MoVoC-Tok, a tokenizer that integrates supervised morphological analysis into the subword vocabulary.
  arXiv  Detail & Related papers  (2025-09-10T17:45:10Z)
• Attention with Trained Embeddings Provably Selects Important Tokens [73.77633297039097]
  Token embeddings play a crucial role in language modeling but, despite this practical relevance, their theoretical understanding remains limited.<n>Our paper addresses the gap by characterizing the structure of embeddings obtained via gradient descent.<n>Experiments on real-world datasets (IMDB, Yelp) exhibit a phenomenology close to that unveiled by our theory.
  arXiv  Detail & Related papers  (2025-05-22T21:00:09Z)
• MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies [0.8029049649310211]
  Tokenization is fundamental to Natural Language Processing (NLP)<n>MorphBPE is a morphology-aware extension of Byte Pair.<n>It integrates linguistic structure into subword tokenization while preserving statistical efficiency.
  arXiv  Detail & Related papers  (2025-02-02T20:06:39Z)
• Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass [72.07642648108849]
  Superposed Decoding is a new decoding algorithm that generates $k$ drafts at the cost of one autoregressive inference pass. Superposed Decoding can be combined with other decoding strategies, resulting in universal coverage gains when scaling inference time compute.
  arXiv  Detail & Related papers  (2024-05-28T17:40:48Z)
• The Role of $n$-gram Smoothing in the Age of Neural Networks [60.23726773548038]
  This paper re-opens the role classical $n$-gram smoothing techniques may play in the age of neural language models. We derive a framework for converting any $n$-gram smoothing technique into a regularizer compatible with neural language models.
  arXiv  Detail & Related papers  (2024-03-25T22:42:19Z)
• Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens [138.36729703589512]
  We show that $n$-gram language models are still relevant in this era of neural large language models (LLMs)<n>This was done by modernizing $n$-gram LMs in two aspects. First, we train them at the same data scale as neural LLMs -- 5 trillion tokens.<n>Second, existing $n$-gram LMs use small $n$ which hinders their performance; we instead allow $n$ to be arbitrarily large, by introducing a new $infty$-gram LM with backoff.
  arXiv  Detail & Related papers  (2024-01-30T19:03:49Z)
• $\ extit{Swap and Predict}$ -- Predicting the Semantic Changes in Words across Corpora by Context Swapping [36.10628959436778]
  We consider the problem of predicting whether a given target word, $w$, changes its meaning between two different text corpora. We propose an unsupervised method that randomly swaps contexts between $mathcalC$ and $mathcalC$. Our method achieves significant performance improvements compared to strong baselines for the English semantic change prediction task.
  arXiv  Detail & Related papers  (2023-10-16T13:39:44Z)
• Nonparametric Masked Language Modeling [113.71921977520864]
  Existing language models (LMs) predict tokens with a softmax over a finite vocabulary. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval.
  arXiv  Detail & Related papers  (2022-12-02T18:10:42Z)
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization [50.16128796194463]
  We propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. We introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level.
  arXiv  Detail & Related papers  (2021-06-23T22:24:14Z)
• Categorical Representation Learning: Morphism is All You Need [0.0]
  We provide a construction for categorical representation learning and introduce the foundations of "$textitcategorifier$" Every object in a dataset $mathcalS$ can be represented as a vector in $mathbbRn$ by an $textitencoding map$ $E: mathcalObj(mathcalS)tomathbbRn$. As a proof of concept, we provide an example of a text translator equipped with our technology, showing that our categorical learning model outperforms the
  arXiv  Detail & Related papers  (2021-03-26T23:47:15Z)
• Pruned Wasserstein Index Generation Model and wigpy Package [0.0]
  I propose a Lasso-based shrinkage method to reduce dimensionality for the vocabulary as a pre-processing step prior to fitting the WIG model. I also provide a textitwigpy module in Python to carry out computation in both flavor.
  arXiv  Detail & Related papers  (2020-03-30T18:26:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.