Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?

• URL: http://arxiv.org/abs/2608.10690v1
• Date: Tue, 11 Aug 2026 09:14:25 GMT
• Title: Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?
• Abstract summary: We first show that BPE tokenizers trained on different corpora share stable token ID-- distributions, motivating distribution transfer from known corpora to a target tokenizer trained on hidden corpora.<n>We then propose Quantile-Guided Density Estimation (QGDE), which approximates this distribution with multiple quantile trends and uses local density weighting to produce token-level estimates.
• Score: 43.96699318814863
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretraining corpus composition shapes LLM capabilities, but it often remains hidden even when model weights are released. Prior work has inferred corpus mixtures or traced specific token groups from released tokenizer vocabularies; in contrast, we estimate corpus ratios for arbitrary target tokens. We first show that BPE tokenizers trained on different corpora share stable token ID--ratio distributions, motivating distribution transfer from known corpora to a target tokenizer trained on hidden corpora. We then propose Quantile-Guided Density Estimation (QGDE), which approximates this distribution with multiple quantile trends and uses local density weighting to produce token-level estimates. In controlled settings and a realistic setting using the released SmolLM tokenizer, QGDE achieves mean relative errors as low as 3.00% for token-level estimation and 3.08% after aggregation into category-level mixtures. These results suggest that released tokenizer vocabularies provide a useful signal for fine-grained corpus estimation beyond coarse composition inference.

Related papers

• Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding [0.0]
  Pruned BPE is a post-training visibility-pruning and token-reallocation method.<n>It separates merge construction from model-visible vocabulary selection.<n>Experiments show that Pruned BPE consistently reduces encoded length relative to Standard BPE.
  arXiv  Detail & Related papers  (2026-08-01T19:34:44Z)
• Text Corpora as Concept Fields: Black-Box Hallucination and Novelty Measurement [3.035872403903575]
  We introduce the textbfConcept Field of a text corpus: a local drift field with pointwise uncertainty.<n>We score its agreement with the field by $$, the mean absolute z-distance between the observed delta and the field's local Gaussian estimate.<n>We evaluate this approach on two large-scale settings: groundedness detection over the U.S. Code of Federal Regulations, and novelty detection over Project Gutenberg.
  arXiv  Detail & Related papers  (2026-05-06T16:38:49Z)
• Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit [45.18582668677648]
  We present a training-free method to transplant tokenizers in large language models.<n>We approximate each out-of-vocabulary token as a sparse linear combination of shared tokens.<n>We show that OMP achieves best zero-shot preservation of the base model's performance.
  arXiv  Detail & Related papers  (2025-06-07T00:51:27Z)
• Self-Supervised and Generalizable Tokenization for CLIP-Based 3D Understanding [87.68271178167373]
  We present a universal 3D tokenizer designed for scale-invariant representation learning with a frozen CLIP backbone.<n>S4Token is a tokenization pipeline that produces semantically-informed tokens regardless of scene scale.
  arXiv  Detail & Related papers  (2025-05-24T18:26:30Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs)<n>We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length.<n>PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method [108.56493934296687]
  We introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection.<n>We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text.
  arXiv  Detail & Related papers  (2024-09-23T07:55:35Z)
• Assessing Keyness using Permutation Tests [0.0]
  We replace the token-by-token sampling model by a model where corpora are samples of documents rather than tokens. We do not need any assumption on how the tokens are organized within or across documents, and the approach works with basically *any* keyness score.
  arXiv  Detail & Related papers  (2023-08-25T13:52:57Z)
• Nonparametric Masked Language Modeling [113.71921977520864]
  Existing language models (LMs) predict tokens with a softmax over a finite vocabulary. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval.
  arXiv  Detail & Related papers  (2022-12-02T18:10:42Z)
• Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages [112.65994041398481]
  We propose a Bayesian generative model for the space of neural parameters. We infer the posteriors over such latent variables based on data from seen task-language combinations. Our model yields comparable or better results than state-of-the-art, zero-shot cross-lingual transfer methods.
  arXiv  Detail & Related papers  (2020-01-30T16:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.