Peek2: A Regex-free implementation of pretokenizers for Byte-level BPE

• URL: http://arxiv.org/abs/2601.05833v1
• Date: Fri, 09 Jan 2026 15:05:09 GMT
• Title: Peek2: A Regex-free implementation of pretokenizers for Byte-level BPE
• Abstract summary: Peek2 is a drop-in replacement for cl100k-like pretokenizers used in GPT-3, LLaMa-3, and Qwen-2.5.<n>It delivers a $ 1.11times $ improvement in overall throughput across the entire Byte-level BPE encoding process.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretokenization is a crucial, sequential pass in Byte-level BPE tokenizers. Our proposed new implementation, Peek2, serves as a drop-in replacement for cl100k-like pretokenizers used in GPT-3, LLaMa-3, and Qwen-2.5. Designed with performance and safety in mind, Peek2 is Regex-free and delivers a $ 1.11\times $ improvement in overall throughput across the entire Byte-level BPE encoding process. This algorithm runs entirely on the CPU, has stable linear complexity $ O(n) $, and provides presegmentation results identical to those of the original Regex-based pretokenizer.

Related papers

• Incremental BPE Tokenization [12.857819598120676]
  We propose a novel algorithm for incremental Byte Pair.<n>Model (BPE) tokenization.<n>We demonstrate significant latency reductions over OpenAI's tiktoken on pathological inputs.
  arXiv  Detail & Related papers  (2026-05-29T04:04:32Z)
• DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts [32.57334288518084]
  Modern RL post-training methods such as GRPO and DAPO train on $N$ response sequences of $R$ tokens sampled from a shared prompt.<n>Standard FlashAttention replicates all $P$ prompt tokens $N times across both forward and backward passes.<n>We propose textDualAttention, the first Flash kernel variant eliminates shared-prompt replication during training.
  arXiv  Detail & Related papers  (2026-05-14T21:11:32Z)
• Leuvenshtein: Efficient FHE-based Edit Distance Computation with Single Bootstrap per Cell [9.554839026375568]
  Edit distance computations are essential in applications across finance and genomics, such as DNA sequence alignment.<n>We introduce an optimised algorithm that significantly reduces the cost of edit distance calculations called Leuvenshtein.
  arXiv  Detail & Related papers  (2025-08-20T09:40:06Z)
• BlockBPE: Parallel BPE Tokenization [0.0]
  BlockBPE is a parallel GPU implementation of byte-pair encoding (BPE)<n>It achieves near linear-time complexity under realistic assumptions.<n>On high-batch inference workloads, BlockBPE achieves up to 2x higher throughput than tiktoken and 2.5x over HuggingFace Tokenizers.
  arXiv  Detail & Related papers  (2025-07-16T06:12:41Z)
• Pre$^3$: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation [32.84102573817514]
  Existing methods parse LR(1) grammars into a pushdown automaton (PDA)<n>We propose Pre$3$ that exploits deterministic pushdown automata (DPDA)<n>Pre$3$ can be seamlessly integrated into standard LLM inference frameworks.
  arXiv  Detail & Related papers  (2025-06-04T12:30:30Z)
• A Partition Cover Approach to Tokenization [24.595558878756787]
  Tokenization is a process of encoding strings into tokens of a fixed vocabulary size.<n>Byte-Pair corpora (BPE) formulates the tokenization problem as a compression problem and tackles it by performing sequences of merges.<n>We show that GreedTok outperforms BPE and Unigram on compression and achieves a covering score comparable to GreedWMC.
  arXiv  Detail & Related papers  (2025-01-08T17:07:07Z)
• HashAttention: Semantic Sparsity for Faster Inference [95.31739930718116]
  This paper introduces HashAttention, framing pivotal token identification as a recommendation problem.<n>It reduces tokens used by up to $16times$ with minimal quality loss, requiring only 32 bits of auxiliary memory per token.<n>On A100 GPU, at $32times$ sparsity, incorporating HashAttention reduces attention latency by up to $4.3times$ in GPT-FAST and $2.54times$ in FlashDecode, and achieves up to $3.12times$ higher throughput for GPT-FAST.
  arXiv  Detail & Related papers  (2024-12-19T02:34:15Z)
• Batching BPE Tokenization Merges [55.2480439325792]
  BatchBPE is an open-source pure Python implementation of the Byte Pair algorithm. It is used to train a high quality tokenizer on a basic laptop.
  arXiv  Detail & Related papers  (2024-08-05T09:37:21Z)
• A Formal Perspective on Byte-Pair Encoding [100.75374173565548]
  Byte-Pair imation (BPE) is a popular algorithm used for tokenizing data in NLP, despite being devised initially as a compression method. We provide a faster implementation of BPE which improves the runtime complexity from $mathcalOleft(N Mright)$ to $mathcalOleft(N log Mright)$, where $N$ is the sequence length and $M$ is the merge count.
  arXiv  Detail & Related papers  (2023-06-29T10:29:23Z)
• Matching Pursuit Based Scheduling for Over-the-Air Federated Learning [67.59503935237676]
  This paper develops a class of low-complexity device scheduling algorithms for over-the-air learning via the method of federated learning. Compared to the state-of-the-art proposed scheme, the proposed scheme poses a drastically lower efficiency system. The efficiency of the proposed scheme is confirmed via experiments on the CIFAR dataset.
  arXiv  Detail & Related papers  (2022-06-14T08:14:14Z)
• 1$\times$N Block Pattern for Network Sparsity [90.43191747596491]
  We propose one novel concept of $1times N$ block sparsity pattern (block pruning) to break this limitation. Our pattern obtains about 3.0% improvements over filter pruning in the top-1 accuracy of MobileNet-V2. It also obtains 56.04ms inference savings on Cortex-A7 CPU over weight pruning.
  arXiv  Detail & Related papers  (2021-05-31T05:50:33Z)
• Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation [80.38621085548013]
  This paper introduces Dynamic Programming (DPE) a new segmentation algorithm for tokenizing sentences into subword units. A mixed character-subword transformer is proposed, which enables exact log marginal likelihood estimation and exact MAP inference to find target segmentations.
  arXiv  Detail & Related papers  (2020-05-03T05:00:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.