Speculating LLMs' Chinese Training Data Pollution from Their Tokens

• URL: http://arxiv.org/abs/2508.17771v2
• Date: Tue, 30 Sep 2025 07:36:58 GMT
• Title: Speculating LLMs' Chinese Training Data Pollution from Their Tokens
• Abstract summary: Polluted Chinese (PoC) tokens are indicated contents like pornography or online gambling.<n>We give a formal definition and taxonomy of PoC tokens based on the GPT's vocabulary.<n>We build a PoC token detector via fine-tuning an LLM to label PoC tokens in vocabularies.
• Score: 59.325920108736454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tokens are basic elements in the datasets for LLM training. It is well-known that many tokens representing Chinese phrases in the vocabulary of GPT (4o/4o-mini/o1/o3/4.5/4.1/o4-mini) are indicating contents like pornography or online gambling. Based on this observation, our goal is to locate Polluted Chinese (PoC) tokens in LLMs and study the relationship between PoC tokens' existence and training data. (1) We give a formal definition and taxonomy of PoC tokens based on the GPT's vocabulary. (2) We build a PoC token detector via fine-tuning an LLM to label PoC tokens in vocabularies by considering each token's both semantics and related contents from the search engines. (3) We study the speculation on the training data pollution via PoC tokens' appearances (token ID). Experiments on GPT and other 23 LLMs indicate that tokens widely exist while GPT's vocabulary behaves the worst: more than 23% long Chinese tokens (i.e., a token with more than two Chinese characters) are either porn or online gambling. We validate the accuracy of our speculation method on famous pre-training datasets like C4 and Pile. Then, considering GPT-4o, we speculate that the ratio of "Yui Hatano" related webpages in GPT-4o's training data is around 0.5%.

Related papers

• TIAO: Token Importance-Aware Policy Optimization for Text Summarization [3.362110408874633]
  We propose Token Importance-Aware Policy Optimization (TIAO), a novel reinforcement learning strategy that explicitly leverages token-importance awareness.<n>TIAO identifies core tokens based on token dependency and reweights a trajectory's advantage according to its overall dependencies.<n>Experiments on the real world dataset show that our TIAO achieves highly competitive results.
  arXiv  Detail & Related papers  (2026-09-15T07:23:07Z)
• TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment [46.819665083185136]
  Inefficient tokenization results in long token-ID sequences and will slow down the training and inference of Large Language Models.<n>We introduce a method named TokAlign++ to improve vocabulary adaptation performance by learning better token alignment lexicon.<n> Experimental results on 15 languages show that our method boosts the multilingual text compression rates and preserves most of the multilingual ability of vanilla models.
  arXiv  Detail & Related papers  (2026-05-13T12:23:24Z)
• Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece [0.0]
  Sanskrit is hypothesized to express more meaning per token due to its morphology and grammar rules.<n>We use a dataset of 701 parallel verses of the Bhagavad Gita, which comprises three languages-Sanskrit, English, and Hindi.<n>We test tokenizers including SentencePiece (SPM), older GPT models, and the latest generation tokenizers from Gemini and GPT.
  arXiv  Detail & Related papers  (2026-01-05T15:23:50Z)
• KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications [13.284214798785847]
  We introduce domain-specific BPE tokenizers for legal, financial, and governmental text.<n>Our kl3m-004-128k-cased tokenizer uses 9-17% fewer tokens than GPT-4o and Llama3 for domain-specific documents.<n>For specialized terminology, our cased tokenizer is even more efficient, using up to 83% fewer tokens for legal terms and 39% fewer tokens for financial terms.
  arXiv  Detail & Related papers  (2025-03-21T15:51:43Z)
• Beyond Literal Token Overlap: Token Alignability for Multilinguality [53.680462160878925]
  We propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation.<n>In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low.<n>We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer.
  arXiv  Detail & Related papers  (2025-02-10T13:50:12Z)
• Token Composition: A Graph Based on EVM Logs [0.02730969268472861]
  We perform an empirical analysis of token composition on the blockchain. We introduce a graph that represents the tokenisation of tokens by other tokens. We relate properties of the graph, e.g., connected components and cyclic structure, to the tokenisation process.
  arXiv  Detail & Related papers  (2024-11-03T21:37:54Z)
• Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data? [112.0422370149713]
  We tackle a task which we call data mixture inference, which aims to uncover the distributional make-up of training data.<n>We introduce a novel attack based on a previously overlooked source of information: byte-pair encoding (BPE) tokenizers.<n>We show that our attack recovers mixture ratios with high precision for tokenizers trained on known mixtures of natural languages, programming languages, and data sources.
  arXiv  Detail & Related papers  (2024-07-23T16:13:22Z)
• Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs [20.1025293763531]
  Llama-2-7b's tokenizer splits the word "northeastern" into the tokens ['_n', 'ort', 'he', 'astern'], none of which correspond to semantically meaningful units like "north" or "east" In this work, we find that last token representations of named entities and multi-token words exhibit a pronounced "erasure" effect, where information about previous and current tokens is rapidly forgotten in early layers.
  arXiv  Detail & Related papers  (2024-06-28T17:54:47Z)
• Identifying and Analyzing Performance-Critical Tokens in Large Language Models [52.404072802235234]
  We study how large language models learn to perform tasks from demonstrations.<n>Our work sheds light on how large language models learn to perform tasks from demonstrations and deepens our understanding of the roles different types of tokens play in large language models.
  arXiv  Detail & Related papers  (2024-01-20T20:55:21Z)
• CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM [7.6361655389908965]
  CharPoet is a Chinese classical poetry generation system based on token-free LLM. Our token-free architecture generates in a character-by-character manner, enabling precise control over the number of characters. CharPoet achieves format accuracy above 0.96, outperforming Jiuge-GPT-2 (0.91) and GPT-4 (0.38)
  arXiv  Detail & Related papers  (2024-01-07T15:00:36Z)
• SHUOWEN-JIEZI: Linguistically Informed Tokenizers For Chinese Language Model Pretraining [48.880840711568425]
  We study the influences of three main factors on the Chinese tokenization for pretrained language models. We propose three kinds of tokenizers: SHUOWEN (meaning Talk Word), the pronunciation-based tokenizers; 2) JIEZI (meaning Solve Character), the glyph-based tokenizers. We find that SHUOWEN and JIEZI tokenizers can generally outperform conventional single-character tokenizers.
  arXiv  Detail & Related papers  (2021-06-01T11:20:02Z)
• Token-level Adaptive Training for Neural Machine Translation [84.69646428587548]
  There exists a token imbalance phenomenon in natural language as different tokens appear with different frequencies. vanilla NMT model usually adopts trivial equal-weighted objectives for target tokens with different frequencies. Low-frequency tokens may carry critical semantic information that will affect the translation quality once they are neglected.
  arXiv  Detail & Related papers  (2020-10-09T05:55:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.