A path to natural language through tokenisation and transformers

• URL: http://arxiv.org/abs/2601.03368v1
• Date: Tue, 06 Jan 2026 19:14:04 GMT
• Title: A path to natural language through tokenisation and transformers
• Abstract summary: We investigate how byte--pair encoding (BPE) transforms corpus statistics.<n>BPE drives token frequencies toward a Zipfian power law while inducing a characteristic growth pattern in empirical entropy.<n>We train language models on corpora tokenised at varying BPE depths, revealing that the model predictive entropies increasingly agree with Zipf-derived predictions.
• Score: 46.05021068704905
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Natural languages exhibit striking regularities in their statistical structure, including notably the emergence of Zipf's and Heaps' laws. Despite this, it remains broadly unclear how these properties relate to the modern tokenisation schemes used in contemporary transformer models. In this note, we analyse the information content (as measured by the Shannon entropy) of various corpora under the assumption of a Zipfian frequency distribution, and derive a closed-form expression for the slot entropy expectation value. We then empirically investigate how byte--pair encoding (BPE) transforms corpus statistics, showing that recursive applications of BPE drive token frequencies toward a Zipfian power law while inducing a characteristic growth pattern in empirical entropy. Utilizing the ability of transformers to learn context dependent token probability distributions, we train language models on corpora tokenised at varying BPE depths, revealing that the model predictive entropies increasingly agree with Zipf-derived predictions as the BPE depth increases. Attention-based diagnostics further indicate that deeper tokenisation reduces local token dependencies, bringing the empirical distribution closer to the weakly dependent (near IID) regime. Together, these results clarify how BPE acts not only as a compression mechanism but also as a statistical transform that reconstructs key informational properties of natural language.

Related papers

• Interior interpretability with attention rollout: contraction and propagation profiles in Transformers [17.72586636966418]
  We introduce emphinterior interpretability, a propagation-based perspective on internal model organization.<n>We interpret rollout as a row-stochastic operator encoding attention-mediated propagation between feature tokens.
  arXiv  Detail & Related papers  (2026-07-24T14:51:28Z)
• Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control [77.8471519867791]
  Empirically, we show that entropy polarity reliably predicts entropy changes.<n>We propose Polarity-Aware Policy Optimization (PAPO), which preserves both polarity branches and implements entropy control through advantage reweighting.
  arXiv  Detail & Related papers  (2026-05-12T08:47:05Z)
• Significance-Gain Pair Encoding for LLMs: A Statistical Alternative to Frequency-Based Subword Merging [0.0]
  Significance-Gain BPE is a drop-in alternative merge criterion that measures cohesion via a z-statistic under an independence null model.<n>It reduces validation and test perplexity by 13% and 12%, respectively, and improves validation and test BPC by about 0.9 to 1.0%.
  arXiv  Detail & Related papers  (2026-02-26T20:29:18Z)
• Entropy-Reservoir Bregman Projection: An Information-Geometric Unification of Model Collapse [3.533187668612022]
  We present EntropyReser Bregman Projection- ERBP, an information-geometric framework that unifies these phenomena.<n>Our theory yields a necessary condition for collapse, (ii) a sufficient condition that guarantees a non-language entropy floor, and (iii) closed-form rates that depend on sample size.
  arXiv  Detail & Related papers  (2025-12-16T19:50:03Z)
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [53.22544362024936]
  Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines.<n>Standard algorithms for learning tokenizers rely on frequency-based objectives.<n>We introduce Parity-aware Byte Pair.<n>We find empirically that Parity-aware BPE leads to more equitable token counts across languages.
  arXiv  Detail & Related papers  (2025-08-06T18:14:43Z)
• Entropy-Driven Pre-Tokenization for Byte-Pair Encoding [4.145560327709288]
  Two entropy-informed pre-tokenization strategies guide BPE segmentation using unsupervised information-theoretic cues.<n>We evaluate both methods on a subset of the PKU dataset and demonstrate substantial improvements in segmentation precision, recall, and F1 score compared to standard BPE.
  arXiv  Detail & Related papers  (2025-06-18T21:25:55Z)
• Bayesian Hierarchical Invariant Prediction [4.182645056052712]
  We propose Bayesian Hierarchical Invariant Prediction (BHIP) reframing Invariant Causal Prediction (ICP) through the lens of Hierarchical Bayes.<n>We leverage the hierarchical structure to explicitly test invariance of causal mechanisms under heterogeneous data, resulting in improved computational scalability for a larger number of predictors compared to ICP.<n>In this paper, we test two sparsity inducing priors: horseshoe and spike-and-slab, both of which allow us a more reliable identification of causal features.
  arXiv  Detail & Related papers  (2025-05-16T13:06:25Z)
• Are We Merely Justifying Results ex Post Facto? Quantifying Explanatory Inversion in Post-Hoc Model Explanations [87.68633031231924]
  Post-hoc explanation methods provide interpretation by attributing predictions to input features.<n>Do these explanations unintentionally reverse the natural relationship between inputs and outputs?<n>We propose Inversion Quantification (IQ), a framework that quantifies the degree to which explanations rely on outputs and deviate from faithful input-output relationships.
  arXiv  Detail & Related papers  (2025-04-11T19:00:12Z)
• Entropy-Lens: The Information Signature of Transformer Computations [14.613982627206884]
  We study the evolution of token-level distributions directly in vocabulary space.<n>We compute the Shannon entropy of each intermediate predicted distribution, yielding one interpretable scalar per layer.<n>We introduce Entropy-Lens, a model-agnostic framework that extracts entropy profiles from frozen, off-the-shelf transformers.
  arXiv  Detail & Related papers  (2025-02-23T13:33:27Z)
• Structural Entropy Guided Probabilistic Coding [52.01765333755793]
  We propose a novel structural entropy-guided probabilistic coding model, named SEPC.<n>We incorporate the relationship between latent variables into the optimization by proposing a structural entropy regularization loss.<n> Experimental results across 12 natural language understanding tasks, including both classification and regression tasks, demonstrate the superior performance of SEPC.
  arXiv  Detail & Related papers  (2024-12-12T00:37:53Z)
• Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Models [20.107727903240065]
  We propose DefinitionEMB to re-construct isotropically distributed and semantics-related token embeddings for encoder-based language models. Our experiments demonstrate the effectiveness of leveraging definitions from Wiktionary to re-construct such embeddings.
  arXiv  Detail & Related papers  (2024-08-02T15:00:05Z)
• Federated Conformal Predictors for Distributed Uncertainty Quantification [83.50609351513886]
  Conformal prediction is emerging as a popular paradigm for providing rigorous uncertainty quantification in machine learning. In this paper, we extend conformal prediction to the federated learning setting. We propose a weaker notion of partial exchangeability, better suited to the FL setting, and use it to develop the Federated Conformal Prediction framework.
  arXiv  Detail & Related papers  (2023-05-27T19:57:27Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.