Examining the Limits of Word2Vec with Toki Pona

• URL: http://arxiv.org/abs/2606.17299v1
• Date: Mon, 15 Jun 2026 21:07:31 GMT
• Title: Examining the Limits of Word2Vec with Toki Pona
• Abstract summary: This study examines whether Word2Vec can successfully capture semantic relationships within an extremely reduced vocabulary.<n>We sourced 1.4 million sentences (7.95 million tokens) from the Toki Pona community for training.<n>Approximately 23% of sentences contain non-Toki Pona tokens such as named entities, loanwords, and neologisms.
• Score: 1.9336815376402718
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Word2Vec's effectiveness at generating semantic embeddings has been widely validated, yet it has been tested almost exclusively on languages with large vocabulary inventories. This study examines whether Word2Vec can successfully capture semantic relationships within an extremely reduced vocabulary using data from Toki Pona, a constructed language with approximately 130 words. We sourced 1.4 million sentences (7.95 million tokens) from the Toki Pona community for training. Approximately 23% of sentences in the corpus contain non-Toki Pona tokens such as named entities, loanwords, and neologisms. To investigate whether this linguistic noise enhances or hinders performance -- a topic rarely addressed in word embedding literature -- we trained two distinct models: one retaining these incidental tokens and another filtering them out completely. Evaluation was conducted using quantitative methods measuring word proximity to semantic category centroids, automated silhouette scores via agglomerative clustering, and qualitative analysis utilizing representational similarity matrices compared against English. The results indicate that while sparse, non-core tokens do not affect the relative structure of the learned embeddings, they actually draw similar words closer together in the vector space. Importantly, Word2Vec's effectiveness depends more on distributional patterns than lexicon size even at this extreme lower bound.

Related papers

• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models [53.01170039144264]
  Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages.<n>Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages?<n>We find that models with overlap outperform models with disjoint vocabularies.
  arXiv  Detail & Related papers  (2025-09-23T07:47:54Z)
• Tokens with Meaning: A Hybrid Tokenization Approach for NLP [0.2826977330147589]
  Tokenization plays a pivotal role in natural language processing (NLP)<n>We introduce a hybrid tokenization framework that combines rule-based morphological analysis with statistical subword segmentation.<n>The method uses phono normalization, root-affix, and a novel algorithm that balances morpheme preservation with vocabulary efficiency.
  arXiv  Detail & Related papers  (2025-08-19T22:17:42Z)
• Morphological evaluation of subwords vocabulary used by BETO language model [0.1638581561083717]
  Subword tokenization algorithms are more efficient and can independently build the necessary vocabulary of words and subwords without human intervention. In previous research, we proposed a method to assess the morphological quality of vocabularies, focusing on the overlap between these vocabularies and the morphemes of a given language. By applying this method to vocabularies created by three subword tokenization algorithms, BPE, Wordpiece, and Unigram, we concluded that these vocabularies generally exhibit very low morphological quality. This evaluation helps clarify the algorithm used by the tokenizer, that is, Wordpiece, given the inconsistencies between the authors' claims
  arXiv  Detail & Related papers  (2024-10-03T08:07:14Z)
• An Evaluation of Sindhi Word Embedding in Semantic Analogies and Downstream Tasks [2.3624125155742064]
  We propose a new word embedding based corpus consisting of more than 61 million words crawled from multiple web resources. We design a preprocessing pipeline for the filtration of unwanted text from crawled data. The cleaned vocabulary is fed to state-of-the-art continuous-bag-of-words, skip-gram, and GloVe word embedding algorithms.
  arXiv  Detail & Related papers  (2024-08-28T11:36:29Z)
• Spoken Word2Vec: Learning Skipgram Embeddings from Speech [0.8901073744693314]
  We show how shallow skipgram-like algorithms fail to encode distributional semantics when the input units are acoustically correlated. We illustrate the potential of an alternative deep end-to-end variant of the model and examine the effects on the resulting embeddings.
  arXiv  Detail & Related papers  (2023-11-15T19:25:29Z)
• Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora [54.757845511368814]
  The problem of comparing two bodies of text and searching for words that differ in their usage arises often in digital humanities and computational social science. This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large. We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word.
  arXiv  Detail & Related papers  (2021-12-28T23:46:00Z)
• Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences [69.3939291118954]
  This paper reports research on a set of comprehensive clustering and network analyses targeting sentence and sub-sentence embedding spaces. Results show that one method generates the most clusterable embeddings. In general, the embeddings of span sub-sentences have better clustering properties than the original sentences.
  arXiv  Detail & Related papers  (2021-10-02T00:47:35Z)
• More Than Words: Collocation Tokenization for Latent Dirichlet Allocation Models [71.42030830910227]
  We propose a new metric for measuring the clustering quality in settings where the models differ. We show that topics trained with merged tokens result in topic keys that are clearer, more coherent, and more effective at distinguishing topics than those unmerged models.
  arXiv  Detail & Related papers  (2021-08-24T14:08:19Z)
• On the Effects of Using word2vec Representations in Neural Networks for Dialogue Act Recognition [0.6767885381740952]
  We propose a new deep neural network that explores recurrent models to capture word sequences within sentences. We validate this model on three languages: English, French and Czech.
  arXiv  Detail & Related papers  (2020-10-22T07:21:17Z)
• Speakers Fill Lexical Semantic Gaps with Context [65.08205006886591]
  We operationalise the lexical ambiguity of a word as the entropy of meanings it can take. We find significant correlations between our estimate of ambiguity and the number of synonyms a word has in WordNet. This suggests that, in the presence of ambiguity, speakers compensate by making contexts more informative.
  arXiv  Detail & Related papers  (2020-10-05T17:19:10Z)
• A Comparative Study on Structural and Semantic Properties of Sentence Embeddings [77.34726150561087]
  We propose a set of experiments using a widely-used large-scale data set for relation extraction. We show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.
  arXiv  Detail & Related papers  (2020-09-23T15:45:32Z)
• Comparative Analysis of Word Embeddings for Capturing Word Similarities [0.0]
  Distributed language representation has become the most widely used technique for language representation in various natural language processing tasks. Most of the natural language processing models that are based on deep learning techniques use already pre-trained distributed word representations, commonly called word embeddings. selecting the appropriate word embeddings is a perplexing task since the projected embedding space is not intuitive to humans.
  arXiv  Detail & Related papers  (2020-05-08T01:16:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.