When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models

• URL: http://arxiv.org/abs/2608.26319v1
• Date: Wed, 26 Aug 2026 18:51:51 GMT
• Title: When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models
• Abstract summary: We study the degradation rate across neural models, both sentence embeddings and decoder-only LLMs.<n>We find that under word-level noise, models with very different architectures decline along nearly the same curve, while under character-level noise they separate.<n>This finding and its underlying mechanism provide a practical means to predict a model's robustness to noise without any noisy evaluation.
• Score: 3.6243102296545846
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The performance of textual neural models often degrades when their inputs are corrupted by noise such as typos, OCR errors, or dropped words. We study the degradation rate across neural models, both sentence embeddings and decoder-only LLMs, and find that how consistent it is depends on the scale of the noise: under word-level noise, models with very different architectures decline along nearly the same curve, while under character-level noise they separate. We further identify the determining factor to be the training objective, not the architecture: eight encoders spanning six pretraining paradigms are scattered initially, and collapse onto a common curve after a short contrastive training recipe. We trace the word/character split to tokenization: a single character edit forces the tokenizer to re-segment the surrounding word, disturbing the token sequence far more than dropping a whole word does. This finding and its underlying mechanism provide a practical means to predict a model's robustness to noise without any noisy evaluation, and to install robustness at a chosen noise scale through noise-augmented training.

Related papers

• Exact learning of quantum noise with tensor networks [0.0]
  We present a variational framework that learns the noise model directly from quantum-error-correction syndrome and logical-observable data.<n>We prove that this objective is principled rather than ad hoc: a sufficiently expressive noise ansatz attains the information-theoretic minimum logical error rate.
  arXiv  Detail & Related papers  (2026-08-31T18:01:16Z)
• From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model [72.73512218682187]
  We introduce ReDiff, a refining-enhanced diffusion framework that teaches the model to identify and correct its own errors.<n>Our approach features a two-stage training process: first, we instill a foundational revision capability by training the model to revise synthetic errors; second, we implement a novel online self-correction loop.<n>This mistake-driven learning endows the model with the crucial ability to revisit and refine its already generated output, effectively breaking the error cascade.
  arXiv  Detail & Related papers  (2025-10-22T06:58:55Z)
• Enhance Vision-Language Alignment with Noise [59.2608298578913]
  We investigate whether the frozen model can be fine-tuned by customized noise.<n>We propose Positive-incentive Noise (PiNI) which can fine-tune CLIP via injecting noise into both visual and text encoders.
  arXiv  Detail & Related papers  (2024-12-14T12:58:15Z)
• NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition [3.726602636064681]
  We present an analysis that shows that real noise is significantly more challenging than simulated noise. We show that current state-of-the-art models for noise-robust learning fall far short of their theoretically achievable upper bound.
  arXiv  Detail & Related papers  (2024-05-13T10:20:31Z)
• Understanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thought [0.0]
  We study how noise in chain of thought impacts task performance in highly-controlled setting. We define two types of noise: textitstatic noise, a local form of noise which is applied after the CoT trace is computed, and textitdynamic noise, a global form of noise which propagates errors in the trace as it is computed. We find fine-tuned models are extremely robust to high levels of static noise but struggle significantly more with lower levels of dynamic noise.
  arXiv  Detail & Related papers  (2024-02-06T13:59:56Z)
• An Investigation of Noise in Morphological Inflection [21.411766936034]
  We investigate the types of noise encountered within a pipeline for truly unsupervised morphological paradigm completion. We compare the effect of different types of noise on multiple state-of-the-art inflection models. We propose a novel character-level masked language modeling (CMLM) pretraining objective and explore its impact on the models' resistance to noise.
  arXiv  Detail & Related papers  (2023-05-26T02:14:34Z)
• Latent Class-Conditional Noise Model [54.56899309997246]
  We introduce a Latent Class-Conditional Noise model (LCCN) to parameterize the noise transition under a Bayesian framework. We then deduce a dynamic label regression method for LCCN, whose Gibbs sampler allows us efficiently infer the latent true labels. Our approach safeguards the stable update of the noise transition, which avoids previous arbitrarily tuning from a mini-batch of samples.
  arXiv  Detail & Related papers  (2023-02-19T15:24:37Z)
• Sources of Noise in Dialogue and How to Deal with Them [63.02707014103651]
  Training dialogue systems often entails dealing with noisy training examples and unexpected user inputs. Despite their prevalence, there currently lacks an accurate survey of dialogue noise. This paper addresses this gap by first constructing a taxonomy of noise encountered by dialogue systems.
  arXiv  Detail & Related papers  (2022-12-06T04:36:32Z)
• Self-Normalized Importance Sampling for Neural Language Modeling [97.96857871187052]
  In this work, we propose self-normalized importance sampling. Compared to our previous work, the criteria considered in this work are self-normalized and there is no need to further conduct a correction step. We show that our proposed self-normalized importance sampling is competitive in both research-oriented and production-oriented automatic speech recognition tasks.
  arXiv  Detail & Related papers  (2021-11-11T16:57:53Z)
• Understanding Model Robustness to User-generated Noisy Texts [2.958690090551675]
  In NLP, model performance often deteriorates with naturally occurring noise, such as spelling errors. We propose to model the errors statistically from grammatical-error-correction corpora.
  arXiv  Detail & Related papers  (2021-10-14T14:54:52Z)
• Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding [76.89426311082927]
  Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference. We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
  arXiv  Detail & Related papers  (2021-04-13T17:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.