Frequency-Ordered Tokenization for Better Text Compression

• URL: http://arxiv.org/abs/2602.22958v1
• Date: Thu, 26 Feb 2026 12:53:48 GMT
• Title: Frequency-Ordered Tokenization for Better Text Compression
• Abstract summary: Method tokenizes text with Byte Pair.<n> encodes the result with variable-length integers before passing it to any standard compressor.<n>On enwik8 (100 MB Wikipedia), this yields improvements of 7.08 percentage points (pp) for zlib, 1.69 pp for LZMA, and 0.76 pp for zstd.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present frequency-ordered tokenization, a simple preprocessing technique that improves lossless text compression by exploiting the power-law frequency distribution of natural language tokens (Zipf's law). The method tokenizes text with Byte Pair Encoding (BPE), reorders the vocabulary so that frequent tokens receive small integer identifiers, and encodes the result with variable-length integers before passing it to any standard compressor. On enwik8 (100 MB Wikipedia), this yields improvements of 7.08 percentage points (pp) for zlib, 1.69 pp for LZMA, and 0.76 pp for zstd (all including vocabulary overhead), outperforming the classical Word Replacing Transform. Gains are consistent at 1 GB scale (enwik9) and across Chinese and Arabic text. We further show that preprocessing accelerates compression for computationally expensive algorithms: the total wall-clock time including preprocessing is 3.1x faster than raw zstd-22 and 2.4x faster than raw LZMA, because the preprocessed input is substantially smaller. The method can be implemented in under 50 lines of code.

Related papers

• Real-Time Text Transmission via LLM-Based Entropy Coding over Fixed-Rate Channels [2.3227078506258176]
  Huffman is the practical choice for over-provisioned channels, with zero algorithmic delay and modest compression overhead.<n>Findings are validated across two scales: GPT-2 (124M) and Llama3.2 (3B), a twenty-five-fold parameter range.
  arXiv  Detail & Related papers  (2026-05-03T17:48:37Z)
• UIPress: Bringing Optical Token Compression to UI-to-Code Generation [2.462537008029033]
  We propose a lightweight learned compression module for UI-to-Code generation.<n> UIPress combines depthwise-separable convolutions, element-guided spatial reweighting, and Transformer refinement to compress visual tokens to a fixed budget of 256.<n>To the best of our knowledge, UIPress is the first encoder-side learned compression method for the UI-to-Code task.
  arXiv  Detail & Related papers  (2026-04-10T15:58:31Z)
• Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exiting [70.75913449565203]
  A Transformer-based encoder has been widely used with block processing.<n>We propose a new encoder, Spiralformer, tailored for block processing by combining layer dropping and early exiting.<n> Experimentally, we observed that our method achieved 21.6% reduction in the averaged token emission delay in Librispeech.
  arXiv  Detail & Related papers  (2025-10-01T14:56:45Z)
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [53.22544362024936]
  Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines.<n>Standard algorithms for learning tokenizers rely on frequency-based objectives.<n>We introduce Parity-aware Byte Pair.<n>We find empirically that Parity-aware BPE leads to more equitable token counts across languages.
  arXiv  Detail & Related papers  (2025-08-06T18:14:43Z)
• CODEPROMPTZIP: Code-specific Prompt Compression for Retrieval-Augmented Generation in Coding Tasks with LMs [6.936336826531964]
  Retrieval-Augmented Generation (RAG) enhances coding tasks by incorporating retrieved code examples into prompts.<n>Existing prompt compression techniques focus on natural language, lacking tailored solutions for code.<n>We propose CodePromptZip, a framework that compresses code examples before integrating into RAG.
  arXiv  Detail & Related papers  (2025-02-19T23:15:23Z)
• Vision-centric Token Compression in Large Language Model [51.92055188780033]
  Vision Centric Token Compression (Vist) is a slow-fast compression framework that mirrors human reading.<n>On eleven in-context learning benchmarks, Vist achieves the same accuracy with 2.3 times fewer tokens, cutting FLOPs by 16% and memory by 50%.
  arXiv  Detail & Related papers  (2025-02-02T13:10:06Z)
• A Partition Cover Approach to Tokenization [24.595558878756787]
  Tokenization is a process of encoding strings into tokens of a fixed vocabulary size.<n>Byte-Pair corpora (BPE) formulates the tokenization problem as a compression problem and tackles it by performing sequences of merges.<n>We show that GreedTok outperforms BPE and Unigram on compression and achieves a covering score comparable to GreedWMC.
  arXiv  Detail & Related papers  (2025-01-08T17:07:07Z)
• Training LLMs over Neurally Compressed Text [55.11828645767342]
  This paper explores the idea of training large language models (LLMs) over highly compressed text.<n>We propose Equal-Info Windows, a novel compression technique whereby text is segmented into blocks that each compress to the same bit length.<n>We demonstrate effective learning over neurally compressed text that improves with scale, and outperforms byte-level baselines by a wide margin on perplexity and inference speed benchmarks.
  arXiv  Detail & Related papers  (2024-04-04T17:48:28Z)
• Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens [65.4435926060951]
  We propose to significantly improve the efficiency of Transformers for ultra long sequences, by compressing the sequence into a much smaller representation at each layer. Our algorithm is not only efficient (achieving more than $3times$ efficiency gain compared to baselines on 4K and 16K lengths) but also offers competitive/better performance on a large number of tasks.
  arXiv  Detail & Related papers  (2023-05-07T10:32:18Z)
• Fast and parallel decoding for transducer [25.510837666148024]
  We introduce a constrained version of transducer loss to learn strictly monotonic alignments between the sequences. We also improve the standard greedy search and beam search algorithms by limiting the number of symbols that can be emitted per time step.
  arXiv  Detail & Related papers  (2022-10-31T07:46:10Z)
• Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding [57.08875260900373]
  We propose Shallow Aggressive Decoding (SAD) to improve the online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC) SAD aggressively decodes as many tokens as possible in parallel instead of always decoding only one token in each step to improve computational parallelism. Experiments in both English and Chinese GEC benchmarks show that aggressive decoding could yield the same predictions but with a significant speedup for online inference.
  arXiv  Detail & Related papers  (2021-06-09T10:30:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.