Explicit Boundary Markers for Subword Vocabularies

• URL: http://arxiv.org/abs/2608.08847v2
• Date: Wed, 12 Aug 2026 07:06:26 GMT
• Title: Explicit Boundary Markers for Subword Vocabularies
• Abstract summary: Subword tokenizers represent many common words twice in space-using writing systems.<n>We introduce an alternative to standard whitespace conventions using an explicit word boundary marker.
• Score: 12.858857146500414
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Subword tokenizers represent many common words twice in space-using writing systems, once with a leading space and once without. The two entries have separate embeddings in models, so occurrences of one word are divided across rows that are trained independently, and the two forms need not even segment the string the same way: " together" may be a single entry while the same word without a preceding space is tokenized as "to|gether". Capitalization divides a word further, into as many as six forms. We introduce an alternative to standard whitespace conventions using an explicit word boundary marker, which prevents such duplication. Words are delimited by the boundary markers, and spaces between words are represented as pairs of such markers. Two shift codes do the same for title case and upper case, allowing one internal representation of a word to be re-used across different settings. Switching to this convention mitigates the duplicate-entry issue, but does not improve tokenization compression: for both vocabulary-learning algorithms, the best marker scheme stays within one percent of the baseline in characters per token, averaged across six languages. It does result in better language modeling performance. Every marker scheme tested downstream reaches lower bits per byte than the baseline, suggesting that duplication carries a cost that compression does not capture.

Related papers

• All You Need Is Non-Commutative Words [0.0]
  We represent lexical tokens as unitary matrices and encode each sentence as their ordered product.<n>The same algebra yields several capabilities, including antisymmetric self-attention with no query, key, or value projections.<n>We achieve higher accuracy on IMDB and comparable performance on AG News.
  arXiv  Detail & Related papers  (2026-08-29T14:56:57Z)
• Vocab Diet: Reshaping the Vocabulary of LLMs with Vector Arithmetic [9.273273023595065]
  Large language models (LLMs) encode word form variations, such as "walk"->"walked", as linear directions in embedding space.<n>Standard tokenization algorithms treat these variations as distinct tokens.<n>We propose a compact reshaping of the vocabulary: rather than assigning unique tokens to each surface form, we compose them from shared base form and transformation vectors.
  arXiv  Detail & Related papers  (2025-10-19T20:56:58Z)
• Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier [11.131061652781192]
  Pre-tokenization causes the distribution of tokens in a corpus to skew towards common, full-length words.<n>We propose BoundlessB, a modified BPE algorithm that relaxes the pretoken boundary constraint.<n>Our approach selectively merges two complete pretokens into a larger unit we term a superword.
  arXiv  Detail & Related papers  (2025-03-31T19:36:29Z)
• A Collocation-based Method for Addressing Challenges in Word-level Metric Differential Privacy [3.0177210416625124]
  Several word-level $textitMetric$ Differential Privacy approaches have been proposed. We devise a method where composed privatized outputs have higher semantic coherence and variable length. We evaluate our method in utility and privacy tests, which make a clear case for tokenization strategies beyond the word level.
  arXiv  Detail & Related papers  (2024-06-30T09:37:34Z)
• DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon [18.05179713472479]
  We introduce DP-Parse, which uses similar principles but only relies on an instance lexicon of word tokens. On the Zero Resource Speech Benchmark 2017, our model sets a new speech segmentation state-of-the-art in 5 languages. Despite lacking a type lexicon, DP-Parse can be pipelined to a language model and learn and semantic representations as assessed by a new spoken word embedding benchmark.
  arXiv  Detail & Related papers  (2022-06-22T19:15:57Z)
• Hierarchical Context Tagging for Utterance Rewriting [51.251400047377324]
  Methods that tag rather than linearly generate sequences have proven stronger in both in- and out-of-domain rewriting settings. We propose a hierarchical context tagger that mitigates this issue by predicting slotted rules. Experiments on several benchmarks show that HCT can outperform state-of-the-art rewriting systems by 2 BLEU points.
  arXiv  Detail & Related papers  (2022-06-22T17:09:34Z)
• MarkBERT: Marking Word Boundaries Improves Chinese BERT [67.53732128091747]
  MarkBERT keeps the vocabulary being Chinese characters and inserts boundary markers between contiguous words. Compared to previous word-based BERT models, MarkBERT achieves better accuracy on text classification, keyword recognition, and semantic similarity tasks.
  arXiv  Detail & Related papers  (2022-03-12T08:43:06Z)
• Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora [54.757845511368814]
  The problem of comparing two bodies of text and searching for words that differ in their usage arises often in digital humanities and computational social science. This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large. We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word.
  arXiv  Detail & Related papers  (2021-12-28T23:46:00Z)
• UniRE: A Unified Label Space for Entity Relation Extraction [67.53850477281058]
  Joint entity relation extraction models setup two separated label spaces for the two sub-tasks. We argue that this setting may hinder the information interaction between entities and relations. In this work, we propose to eliminate the different treatment on the two sub-tasks' label spaces.
  arXiv  Detail & Related papers  (2021-07-09T08:09:37Z)
• Multi-view Subword Regularization [111.04350390045705]
  Multi-view Subword Regularization (MVR) is a method that enforces the consistency between predictions of using inputs tokenized by the standard and probabilistic segmentations. Results on the XTREME multilingual benchmark show that MVR brings consistent improvements of up to 2.5 points over using standard segmentation algorithms.
  arXiv  Detail & Related papers  (2021-03-15T16:07:42Z)
• Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality [24.80654159288458]
  We propose a character-based subword module (char2subword) that learns the subword embedding table in pre-trained models like BERT. Our module is robust to character-level alterations such as misspellings, word inflection, casing, and punctuation. We show that incorporating our module to mBERT significantly improves the performance on the social media linguistic code-switching evaluation (LinCE) benchmark.
  arXiv  Detail & Related papers  (2020-10-24T01:08:28Z)
• 2kenize: Tying Subword Sequences for Chinese Script Conversion [54.33749520569979]
  We propose a model that can disambiguate between mappings and convert between the two scripts. Our proposed method outperforms previous Chinese Character conversion approaches by 6 points in accuracy.
  arXiv  Detail & Related papers  (2020-05-07T10:53:05Z)
• Discovering linguistic (ir)regularities in word embeddings through max-margin separating hyperplanes [0.0]
  We show new methods for learning how related words are positioned relative to each other in word embedding spaces. Our model, SVMCos, is robust to a range of experimental choices when training word embeddings.
  arXiv  Detail & Related papers  (2020-03-07T20:21:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.