X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

• URL: http://arxiv.org/abs/2605.21699v1
• Date: Wed, 20 May 2026 19:59:31 GMT
• Title: X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
• Abstract summary: Cross-tokenizer knowledge distillation allows a student model to learn from teachers with vocabularies.<n>Logit-based methods either use only the correct-token probability, missing the full 'dark knowledge' in the teacher's distribution.<n>We propose X-Token, an approach with two complementary loss formulations targeting these issues.
• Score: 19.064775527374618
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-tokenizer knowledge distillation allows a student model to learn from teachers with incompatible vocabularies. Prior work operates on hidden states or logits; the latter is preferred as a drop-in replacement requiring no auxiliary components. Logit-based methods either use only the correct-token probability, missing the full 'dark knowledge' in the teacher's distribution, or operate on the full output distribution, relying on strict token partitioning and/or unprincipled heuristic ranking. We identify two key shortcomings of full-distribution, logit-based methods: (i) an uncommon-token failure, where critical tokens fall into the unmatched subset (e.g., Llama's 1100 multi-digit numerals under digit-splitting Qwen supervision) and are suppressed during training, reducing GSM8k from 12.89 to 2.56 compared to same-tokenizer KD from a weaker teacher; and (ii) over-conservative matching, where strict 1-to-1 matching excludes near-equivalent tokens across surface forms. These failures require distinct remedies: eliminating the partition when critical tokens are misaligned, and refining it when alignment is reliable. We propose X-Token, an approach with two complementary loss formulations targeting these issues. P-KL removes partitioning and aligns the student's distribution with the teacher's via a sparse projection matrix W (initialized from tokenizer-level string rules) to address the uncommon-token failure. H-KL retains the hybrid form while relaxing matching to align each student token with its top-ranked teacher mapping under W. Both objectives share W and extend naturally to multiple teachers. Empirically, on Llama-3.2-1B, X-Token outperforms the current state of the art GOLD by +3.82 average points with a Qwen3-4B teacher and by +0.5 with a Phi-4-Mini teacher. Further, a two-teacher setup (Phi-4-mini + Llama-3B) improves over single-teacher distillation by +1.3 points.

Related papers

• ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models [4.5311655360445515]
  Local selection can miss tokens that the student considers likely, while student-only selection can rely on an inaccurate ranking early in training.<n>We propose Adaptive Local Alignment (ALRA), a position-specific framework combining student proposals with teacher guidance.<n>ALRA exceeds the strongest competing distillation baseline by 0.94 and 0.83 percentage points and improves over pre-training distillation by 2.31 and 2.91 points, respectively.
  arXiv  Detail & Related papers  (2026-09-03T04:26:39Z)
• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation [59.43605629311855]
  We analyze the gradient of the per-token K2 estimator of reverse KL with respect to the student logits.<n>Our primary contribution is therefore a gradient-level characterization of reverse-KL OPD trained with the K2 estimator, with SuRe as one empirical instantiation.
  arXiv  Detail & Related papers  (2026-08-26T11:14:42Z)
• SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning [54.402550664505576]
  On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models to short-context students.<n>We study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models.<n> Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning.
  arXiv  Detail & Related papers  (2026-08-14T12:57:02Z)
• Mismatch Matters: On-Policy Distillation Beyond Token Agreement [27.396104103072844]
  We show that mismatch tokens can be mainly categorized into two types: student-excess tokens and student-deficit tokens.<n>We propose TIDE, which applies bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass.
  arXiv  Detail & Related papers  (2026-08-10T16:53:14Z)
• OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification [21.095025092772257]
  On-Policy Distillation (OPD) trains a student model on its own generative trajectories under dense token-level feedback from a stronger teacher.<n>We introduce OmniOPD, a novel framework that addresses both limitations through a logit-free, chunk-level supervision signal.
  arXiv  Detail & Related papers  (2026-05-31T22:31:15Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works [84.74671323132492]
  We show that mixing hard and soft labels consistently yields better results.<n>It comes from reduced exposure bias, the mismatch between training and inference distributions.<n>We develop a family of Bridge-Garden hybrid supervision methods.
  arXiv  Detail & Related papers  (2026-05-25T18:12:54Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation [39.59759394378816]
  On-policy distillation assumes teacher and student predictions are comparable token by token.<n>We propose SimCT, which restores this signal by enlarging the supervision space.<n>SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines.
  arXiv  Detail & Related papers  (2026-05-08T13:16:17Z)
• Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes [31.95045602299568]
  On-policy distillation (OPD) is appealing for large language model (LLM) post-training because it evaluates teacher feedback on student-generated rollouts rather than fixed teacher traces.<n>We revisit OPD from the estimator and implementation sides.<n>We identify three failure modes of sampled-token OPD: an imbalanced one-token signal, unreliable teacher guidance on student-generated prefixes, and distortions caused by tokenizer or special-token mismatch.
  arXiv  Detail & Related papers  (2026-03-26T15:35:59Z)
• A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone [41.9829555690118]
  Low-Rank Clone (LRC) is an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models.<n>LRC matches or surpasses state-of-the-art models trained on trillions of tokens--while using only 20B tokens, achieving over 1,000x training efficiency.
  arXiv  Detail & Related papers  (2025-05-19T07:10:42Z)
• Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching [16.385782508179364]
  Distillation has shown remarkable success in transferring knowledge from a Large Language Model (LLM) teacher to a student LLM.<n>Current distillation methods require similar tokenizers between the teacher and the student, restricting their applicability to only a small subset of teacher-student pairs.<n>We develop a principled cross-tokenizer distillation method to solve this crucial deficiency.<n>Our method is the first to enable effective distillation across fundamentally different tokenizers, while also substantially outperforming prior methods in all other cases.
  arXiv  Detail & Related papers  (2025-03-25T21:44:10Z)
• Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping [85.48043537327258]
  Contextual Dynamic Mapping (CDM) is a novel cross-tokenizer distillation framework.<n>It uses contextual information to enhance sequence alignment precision and dynamically improve vocabulary mapping.<n>Our method shows significant advantages over existing cross-tokenizer distillation baselines across diverse benchmarks.
  arXiv  Detail & Related papers  (2025-02-16T12:46:07Z)
• Rho-1: Not All Tokens Are What You Need [132.31428897792114]
  Previous language model pre-training methods uniformly applied a next-token prediction loss to all training tokens.<n>Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that align with desired distribution.<n>When continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks.
  arXiv  Detail & Related papers  (2024-04-11T17:52:01Z)
• JointMatch: A Unified Approach for Diverse and Collaborative Pseudo-Labeling to Semi-Supervised Text Classification [65.268245109828]
  Semi-supervised text classification (SSTC) has gained increasing attention due to its ability to leverage unlabeled data. Existing approaches based on pseudo-labeling suffer from the issues of pseudo-label bias and error accumulation. We propose JointMatch, a holistic approach for SSTC that addresses these challenges by unifying ideas from recent semi-supervised learning.
  arXiv  Detail & Related papers  (2023-10-23T05:43:35Z)
• Shrinking Class Space for Enhanced Certainty in Semi-Supervised Learning [59.44422468242455]
  We propose a novel method dubbed ShrinkMatch to learn uncertain samples. For each uncertain sample, it adaptively seeks a shrunk class space, which merely contains the original top-1 class. We then impose a consistency regularization between a pair of strongly and weakly augmented samples in the shrunk space to strive for discriminative representations.
  arXiv  Detail & Related papers  (2023-08-13T14:05:24Z)
• Fair and skill-diverse student group formation via constrained k-way graph partitioning [65.29889537564455]
  This work introduces an unsupervised algorithm for fair and skill-diverse student group formation. The skill sets of students are determined using unsupervised dimensionality reduction of course mark data via the Laplacian eigenmap. The problem is formulated as a constrained graph partitioning problem, whereby the diversity of skill sets in each group are maximised.
  arXiv  Detail & Related papers  (2023-01-12T14:02:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.