When Informal Text Breaks NLI: Tokenization Failure, Distribution Shift, and Targeted Mitigations

• URL: http://arxiv.org/abs/2604.16787v1
• Date: Sat, 18 Apr 2026 02:23:02 GMT
• Title: When Informal Text Breaks NLI: Tokenization Failure, Distribution Shift, and Targeted Mitigations
• Abstract summary: We study how informal surface forms degrade NLI accuracy across four transforms applied to SNLI and MultiNLI.<n>Emoji replaces content words with Unicode characters that ELECTRA's WordPiece tokenizer maps to [UNK], destroying the input signal before any learned parameters see it.<n>Noise tokens (no cap, deadass, tbh) are fully in-vocabulary but absent from NLI training data, consistent with the model assigning them inferential weight they do not carry.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study how informal surface forms degrade NLI accuracy in ELECTRA-small (14M) and RoBERTa-large (355M) across four transforms applied to SNLI and MultiNLI: slang substitution, emoji replacement, Gen-Z filler tokens, and their combination. Slang substitution (replacing formal words with informal equivalents, e.g., "going to" -> "gonna", "friend" -> "homie") causes minimal degradation (at most 1.1pp): slang vocabulary falls largely within WordPiece coverage, so the tokenizer handles it without signal loss. Emoji replaces content words with Unicode characters that ELECTRA's WordPiece tokenizer maps to [UNK], destroying the input signal before any learned parameters see it (93.6% of emoji examples contain at least one [UNK], mean 2.91 per example). Noise tokens (no cap, deadass, tbh) are fully in-vocabulary but absent from NLI training data, consistent with the model assigning them inferential weight they do not carry. The two failure modes respond to different interventions: preprocessing recovers emoji accuracy by normalizing text before tokenization; augmentation handles noise by exposing the model to noise-bearing examples during training. A hybrid of both achieves 88.93% on the combined variant for ELECTRA on SNLI (up from 75.88%), with no statistically significant drop on clean text. Against GPT-4o-mini zero-shot, unmitigated ELECTRA is significantly worse on transformed variants (p < 0.0001); hybrid ELECTRA surpasses it across all SNLI variants and reaches statistical parity on MultiNLI.

Related papers

• Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility [0.0]
  In abugida scripts, vowels are written as combining marks; this pattern therefore splits each word at every vowel sign.<n>Since BPE merges only within a pre-token, those splits persist through training regardless of vocabulary size or corpus composition.<n>Across 26 languages from a parallel corpus, every one of the 17 abugidas is affected, ranging from 1.47x (Tibetan) to 9.02x (Thai)<n>For 5 languages, matched tokenizer pairs that differ only in this character class fall within 2.2% of the predicted floor, scoring 4.78 versus 1.58 tokens per word on Nepali.
  arXiv  Detail & Related papers  (2026-08-26T22:56:24Z)
• Content is What Remains: Invariant Speech Tokenization from Parallel Utterances [8.166364251367627]
  PINT fine-tunes an SSL encoder with alignment losses across parallel utterances and augmentations to distill this shared residual.<n>Unlike ASR text, PINT tokens preserve frame-level temporal grounding and serve as drop-in semantic targets for audio codecs.
  arXiv  Detail & Related papers  (2026-07-21T12:19:38Z)
• TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment [46.819665083185136]
  Inefficient tokenization results in long token-ID sequences and will slow down the training and inference of Large Language Models.<n>We introduce a method named TokAlign++ to improve vocabulary adaptation performance by learning better token alignment lexicon.<n> Experimental results on 15 languages show that our method boosts the multilingual text compression rates and preserves most of the multilingual ability of vanilla models.
  arXiv  Detail & Related papers  (2026-05-13T12:23:24Z)
• Mind the Pause: Disfluency-Aware Objective Tuning for Multilingual Speech Correction with LLMs [32.94544776067079]
  ASR transcripts often contain disfluencies, such as fillers, repetitions, and false starts.<n>Most existing approaches rely on classical models that focus on identifying disfluent tokens for removal.<n>We propose a multilingual correction pipeline where a sequence tagger first marks disfluent tokens, and these signals guide instruction fine-tuning of an LLM to rewrite transcripts into fluent text.
  arXiv  Detail & Related papers  (2026-05-12T15:11:36Z)
• In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification [4.738949927143789]
  We show that homogeneous labels--even semantically valid ones--collapse accuracy to =12% across six models.<n>The trigger is label-slot content: the model treats tokens occupying the label position as an exhaustive answer vocabulary.
  arXiv  Detail & Related papers  (2026-05-08T10:20:39Z)
• M3T: Discrete Multi-Modal Motion Tokens for Sign Language Production [56.171224102170015]
  Non-manual features, including mouthings, eyebrow raises, gaze, and head movements, are grammatically obligatory and cannot be recovered from manual articulators alone.<n>Existing 3D production systems face two barriers to integrating them: the standard body model provides a facial space too low-dimensional to encode these articulations.<n>We propose SMPL-FX, which couples FLAME's rich expression space with the SMPL-X body, and tokenize the resulting representation with modality-specific Finite Scalar Quantization VAEs for body, hands, and face.
  arXiv  Detail & Related papers  (2026-03-24T18:05:03Z)
• LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers [76.59130257385826]
  Intermediate merge residues in BPE vocabularies are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage.<n>We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens.<n>Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.
  arXiv  Detail & Related papers  (2026-02-04T16:19:05Z)
• Universal Adversarial Suffixes Using Calibrated Gumbel-Softmax Relaxation [9.099589602551573]
  We study universal adversarial suffixes that, when appended to any input, broadly reduce accuracy across tasks and models.<n>Our approach learns the suffix in a differentiable "soft" form using Gumbel-Softmax relaxation and then discretizes it for inference.<n>A single suffix trained on one model transfers effectively to others, consistently lowering both accuracy and calibrated confidence.
  arXiv  Detail & Related papers  (2025-12-09T00:03:39Z)
• Tokens with Meaning: A Hybrid Tokenization Approach for NLP [0.2826977330147589]
  Tokenization plays a pivotal role in natural language processing (NLP)<n>We introduce a hybrid tokenization framework that combines rule-based morphological analysis with statistical subword segmentation.<n>The method uses phono normalization, root-affix, and a novel algorithm that balances morpheme preservation with vocabulary efficiency.
  arXiv  Detail & Related papers  (2025-08-19T22:17:42Z)
• Small Edits, Big Consequences: Telling Good from Bad Robustness in Large Language Models [0.0]
  Large language models (LLMs) now write code in settings where misreading a single word can break safety or cost money, yet we still expect them to overlook stray typos.<n>To probe where useful robustness ends and harmful insensitivity begins, we compile 50 LeetCode problems and craft three minimal prompt perturbations.<n>Six frontier models, including three "reasoning-tuned" versions, solve each mutated prompt.
  arXiv  Detail & Related papers  (2025-07-15T03:22:07Z)
• Sampling from Your Language Model One Byte at a Time [82.71473348639489]
  Tokenization can introduce distortion into the model's generations, known as the Prompt Boundary Problem (PBP)<n>We present an inference-time method to convert any autore LM with a BPE tokenizer into a character-level or byte-level LM.<n>Our method efficiently solves the PBP and is also able to unify the vocabularies of language models with different tokenizers.
  arXiv  Detail & Related papers  (2025-06-17T02:37:04Z)
• TokAlign: Efficient Vocabulary Adaptation via Token Alignment [41.59130966729569]
  Tokenization serves as a foundational step for Large Language Models (LLMs) to process text.<n>In new domains or languages, the inefficiency of the tokenizer will slow down the training and generation of LLM.<n>We propose an efficient method named TokAlign to replace the vocabulary of LLM from the token co-occurrences view.
  arXiv  Detail & Related papers  (2025-06-04T03:15:57Z)
• SuperBPE: Space Travel for Language Models [103.09169510391972]
  We introduce a "superword" tokenizer, SuperBPE, which incorporates a simple pretokenization curriculum into the byte-pair encoding (BPE) algorithm.<n>SuperBPE encodes a fixed piece of text with up to 33% fewer tokens than BPE on average.<n>Our model trained with SuperBPE achieves an average +4.0% absolute improvement over the BPE baseline across 30 downstream tasks.
  arXiv  Detail & Related papers  (2025-03-17T17:53:23Z)
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis [84.57932472551889]
  RALL-E is a robust language modeling method for text-to-speech synthesis. RALL-E improves the WER of zero-shot TTS from $5.6%$ (without reranking) to $2.5%$ and $1.0%$, respectively.
  arXiv  Detail & Related papers  (2024-04-04T05:15:07Z)
• Nonparametric Masked Language Modeling [113.71921977520864]
  Existing language models (LMs) predict tokens with a softmax over a finite vocabulary. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval.
  arXiv  Detail & Related papers  (2022-12-02T18:10:42Z)
• A Token-level Contrastive Framework for Sign Language Translation [9.185037439012952]
  Sign Language Translation is a promising technology to bridge the communication gap between the deaf and the hearing people. We propose ConSLT, a novel token-level. textbfContrastive learning framework for textbfSign textbfLanguage. textbfTranslation.
  arXiv  Detail & Related papers  (2022-04-11T07:33:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.