Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP

• URL: http://arxiv.org/abs/2605.13436v1
• Date: Wed, 13 May 2026 12:31:04 GMT
• Title: Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP
• Abstract summary: Subword regularization methods such as BPE dropout are applied only during fine-tuning, while pretraining is usually done with deterministic tokenization.<n>This creates a potential segmentation mismatch between pretraining and fine-tuning.<n>We train monolingual and bilingual BERT models on downsampled subsets of English, German, French, Spanish, Kiswahili, and isiXhosa, and evaluate them on XNLI, PAWS-X, PAN-X, and MasakhaNER 2.0.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Subword regularization methods such as BPE dropout are typically applied only during fine-tuning, while pretraining is usually done with deterministic tokenization. This creates a potential segmentation mismatch between pretraining and fine-tuning. We investigate whether applying BPE dropout during pretraining improves downstream performance in low-resource NLP. We train monolingual and bilingual BERT models on downsampled subsets of English, German, French, Spanish, Kiswahili, and isiXhosa, and evaluate them on XNLI, PAWS-X, PAN-X, and MasakhaNER 2.0. Across tasks, the best results are typically obtained when stochastic tokenization is applied during both pretraining and fine-tuning, whereas applying BPE dropout only during fine-tuning can underperform deterministic tokenization in smaller-data settings. This disadvantage diminishes as fine-tuning data increases, while the benefits of pretraining-time BPE dropout are largest when either pretraining or fine-tuning data is scarce. The benefits of BPE dropout are often attributed to better compositional representations, especially for rare words. To examine this, we measure morphological boundary alignment under BPE dropout and find only modest improvements in expected alignment, while better-aligned segmentations remain rare. This suggests that fine-tuning alone may provide limited exposure to such segmentations, whereas stochastic tokenization during pretraining exposes the model to them more consistently. We further show that selectively introducing morphologically aligned segmentations during fine-tuning improves performance mainly for models pretrained without BPE dropout. Overall, these findings suggest that exposure to better-aligned segmentations may contribute to the downstream benefits of applying BPE dropout during pretraining.

Related papers

• Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data [32.68918472543787]
  Large language models rely on web-scale corpora for pre-training.<n>Data curation mitigates but cannot eliminate such noise, so pre-training corpora remain noisy in practice.<n>We study whether a lightweight pre-pre-training stage helps resist noisy data during the pre-training stage.
  arXiv  Detail & Related papers  (2026-05-11T07:40:39Z)
• The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data [55.87500250831868]
  We study a simple strategy, specialized pretraining (SPT), where a small domain dataset is repeated starting from pretraining as a fraction of the total tokens.<n>In our experiments, SPT reduces the pretraining tokens needed to reach a given domain performance by up to 1.75x.<n>Our observations reveal the finetuner's fallacy: while finetuning may appear to be the cheapest path to domain adaptation, introducing specialized domain data during pretraining stretches its utility.
  arXiv  Detail & Related papers  (2026-03-17T06:55:53Z)
• When Priors Backfire: On the Vulnerability of Unlearnable Examples to Pretraining [28.704214093822472]
  Unlearnable Examples (UEs) serve as a data protection strategy that generates imperceptible perturbations to mislead models into learning spurious correlations instead of underlying semantics.<n>In this paper, we uncover a fundamental vulnerability of UEs that emerges when learning starts from a pretrained model.<n>We propose BAIT (Binding Artificial perturbations to Incorrect Targets), a novel bi-level optimization formulation.
  arXiv  Detail & Related papers  (2026-03-05T02:10:40Z)
• Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings [14.342802193320365]
  Dropping the Positional Embeddings of LMs after training (DroPE) is motivated by three key theoretical and empirical observations.<n> positional embeddings (PEs) serve a crucial role during pretraining, providing an important inductive bias.<n>DroPE yields seamless zero-shot context extension without any long-context finetuning.
  arXiv  Detail & Related papers  (2025-12-13T04:23:47Z)
• Midtraining Bridges Pretraining and Posttraining Distributions [73.84346031272473]
  "Midtraining" is a phase in which higher quality, often instruction-formatted data is mixed in at the end of pretraining.<n>We conduct the first systematic investigation of midtraining through experiments with language models pretrained from scratch.<n>We find that when compared after supervised fine-tuning, the effectiveness of midtraining is highest in the math and code domains.
  arXiv  Detail & Related papers  (2025-10-16T16:39:52Z)
• Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training [57.03005244917803]
  Large language models (LLMs) often fail on out-of-distribution (OOD) samples due to spurious correlations acquired during pre-training.<n>Here, we aim to mitigate such spurious correlations through causality-aware post-training (CAPT)<n> Experiments on the formal causal inference benchmark CLadder and the logical reasoning dataset PrOntoQA show that 3B-scale language models fine-tuned with CAPT can outperform both traditional SFT and larger LLMs on in-distribution (ID) and OOD tasks.
  arXiv  Detail & Related papers  (2025-06-11T06:30:28Z)
• ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining [53.893792844055106]
  Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency.<n>We introduce Selective Efficient Language Modeling, a risk-aware algorithm that improves training efficiency and distributional robustness by performing online token-level batch selection.<n> Experiments on GPT-2 pretraining show that ESLM significantly reduces training FLOPs while maintaining or improving both perplexity and downstream performance compared to baselines.
  arXiv  Detail & Related papers  (2025-05-26T12:23:26Z)
• Overtrained Language Models Are Harder to Fine-Tune [64.44743256512237]
  Large language models are pre-trained on ever-growing token budgets.<n>We show that extended pre-training can make models harder to fine-tune, leading to degraded final performance.
  arXiv  Detail & Related papers  (2025-03-24T23:11:56Z)
• Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models [26.442558912559658]
  We show a fundamental limitation in vocabulary adaptation approaches that use Byte-Pair. (BPE) tokenization scheme for fine-tuning pretrained language models (PLMs) to expert domains. We propose AdaptBPE where the BPE tokenization phase is modified to first perform the longest string matching on the added (target) vocabulary before tokenizing at the character level.
  arXiv  Detail & Related papers  (2024-10-04T09:24:55Z)
• Toward Understanding BERT-Like Pre-Training for DNA Foundation Models [78.48760388079523]
  Existing pre-training methods for DNA sequences rely on direct adoptions of BERT pre-training from NLP. We introduce a novel approach called RandomMask, which gradually increases the task difficulty of BERT-like pre-training by continuously expanding its mask boundary. RandomMask achieves a staggering 68.16% in Matthew's correlation coefficient for Epigenetic Mark Prediction, a groundbreaking increase of 19.85% over the baseline.
  arXiv  Detail & Related papers  (2023-10-11T16:40:57Z)
• Examining the Effect of Pre-training on Time Series Classification [21.38211396933795]
  This study investigates the impact of pre-training followed by fine-tuning on the fine-tuning process. We conducted a thorough examination of 150 classification datasets. We find that pre-training can only help improve the optimization process for models that fit the data poorly. Adding more pre-training data does not improve generalization, but it can strengthen the advantage of pre-training on the original data volume.
  arXiv  Detail & Related papers  (2023-09-11T06:26:57Z)
• Sample-Efficient Optimisation with Probabilistic Transformer Surrogates [66.98962321504085]
  This paper investigates the feasibility of employing state-of-the-art probabilistic transformers in Bayesian optimisation. We observe two drawbacks stemming from their training procedure and loss definition, hindering their direct deployment as proxies in black-box optimisation. We introduce two components: 1) a BO-tailored training prior supporting non-uniformly distributed points, and 2) a novel approximate posterior regulariser trading-off accuracy and input sensitivity to filter favourable stationary points for improved predictive performance.
  arXiv  Detail & Related papers  (2022-05-27T11:13:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.