SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation

• URL: http://arxiv.org/abs/2605.07711v1
• Date: Fri, 08 May 2026 13:16:17 GMT
• Title: SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
• Abstract summary: On-policy distillation assumes teacher and student predictions are comparable token by token.<n>We propose SimCT, which restores this signal by enlarging the supervision space.<n>SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines.
• Score: 39.59759394378816
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose \textbf{\underline{Sim}ple \underline{C}ross-\underline{T}okenizer OPD (SimCT)}, which restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-token matching. Code is available at \href{https://github.com/sunjie279/SimCT-}{https://github.com/sunjie279/SimCT-}.

Related papers

• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation [59.43605629311855]
  We analyze the gradient of the per-token K2 estimator of reverse KL with respect to the student logits.<n>Our primary contribution is therefore a gradient-level characterization of reverse-KL OPD trained with the K2 estimator, with SuRe as one empirical instantiation.
  arXiv  Detail & Related papers  (2026-08-26T11:14:42Z)
• SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning [54.402550664505576]
  On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models to short-context students.<n>We study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models.<n> Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning.
  arXiv  Detail & Related papers  (2026-08-14T12:57:02Z)
• Mismatch Matters: On-Policy Distillation Beyond Token Agreement [27.396104103072844]
  We show that mismatch tokens can be mainly categorized into two types: student-excess tokens and student-deficit tokens.<n>We propose TIDE, which applies bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass.
  arXiv  Detail & Related papers  (2026-08-10T16:53:14Z)
• Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization [15.613307729537096]
  Byte-Prefix Marginalization re-expresses the teacher's next-token distribution over the student vocabulary in a shared byte space.<n>BPM consistently outperforms current cross-tokenizer methods on six mathematics and programming benchmarks.
  arXiv  Detail & Related papers  (2026-07-24T14:12:54Z)
• OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification [21.095025092772257]
  On-Policy Distillation (OPD) trains a student model on its own generative trajectories under dense token-level feedback from a stronger teacher.<n>We introduce OmniOPD, a novel framework that addresses both limitations through a logit-free, chunk-level supervision signal.
  arXiv  Detail & Related papers  (2026-05-31T22:31:15Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation [19.064775527374618]
  Cross-tokenizer knowledge distillation allows a student model to learn from teachers with vocabularies.<n>Logit-based methods either use only the correct-token probability, missing the full 'dark knowledge' in the teacher's distribution.<n>We propose X-Token, an approach with two complementary loss formulations targeting these issues.
  arXiv  Detail & Related papers  (2026-05-20T19:59:31Z)
• PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization [8.126475706520063]
  PairAlign is a framework for compact audio tokenization through sequence-level self-alignment.<n>It learns compact, non-degenerate sequences with broad vocabulary usage.<n>It preserves edit-distance search while reducing archive token count by 55%.
  arXiv  Detail & Related papers  (2026-05-07T17:11:22Z)
• Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes [31.95045602299568]
  On-policy distillation (OPD) is appealing for large language model (LLM) post-training because it evaluates teacher feedback on student-generated rollouts rather than fixed teacher traces.<n>We revisit OPD from the estimator and implementation sides.<n>We identify three failure modes of sampled-token OPD: an imbalanced one-token signal, unreliable teacher guidance on student-generated prefixes, and distortions caused by tokenizer or special-token mismatch.
  arXiv  Detail & Related papers  (2026-03-26T15:35:59Z)
• LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers [76.59130257385826]
  Intermediate merge residues in BPE vocabularies are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage.<n>We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens.<n>Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.
  arXiv  Detail & Related papers  (2026-02-04T16:19:05Z)
• Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit [45.18582668677648]
  We present a training-free method to transplant tokenizers in large language models.<n>We approximate each out-of-vocabulary token as a sparse linear combination of shared tokens.<n>We show that OMP achieves best zero-shot preservation of the base model's performance.
  arXiv  Detail & Related papers  (2025-06-07T00:51:27Z)
• Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping [85.48043537327258]
  Contextual Dynamic Mapping (CDM) is a novel cross-tokenizer distillation framework.<n>It uses contextual information to enhance sequence alignment precision and dynamically improve vocabulary mapping.<n>Our method shows significant advantages over existing cross-tokenizer distillation baselines across diverse benchmarks.
  arXiv  Detail & Related papers  (2025-02-16T12:46:07Z)
• Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation [43.09801987385207]
  Contrastive Language-Image Pretraining (CLIP) excels at learning generalizable image representations but often falls short in zero-shot inference on certain datasets.<n>Test-time adaptation (TTA) mitigates this issue by adjusting components like normalization layers or context prompts, yet it typically requires large batch sizes and extensive augmentations.<n>We propose Token Condensation as Adaptation (TCA), a training-free adaptation method that takes a step beyond standard TC.
  arXiv  Detail & Related papers  (2024-10-16T07:13:35Z)
• Match me if you can: Semi-Supervised Semantic Correspondence Learning with Unpaired Images [76.47980643420375]
  This paper builds on the hypothesis that there is an inherent data-hungry matter in learning semantic correspondences. We demonstrate a simple machine annotator reliably enriches paired key points via machine supervision. Our models surpass current state-of-the-art models on semantic correspondence learning benchmarks like SPair-71k, PF-PASCAL, and PF-WILLOW.
  arXiv  Detail & Related papers  (2023-11-30T13:22:15Z)
• A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings [28.046786376565123]
  We propose a semantics-aware contrastive learning framework for sentence embeddings, termed Pseudo-Token BERT (PT-BERT) We exploit the pseudo-token space (i.e., latent semantic space) representation of a sentence while eliminating the impact of superficial features such as sentence length and syntax. Our model outperforms the state-of-the-art baselines on six standard semantic textual similarity (STS) tasks.
  arXiv  Detail & Related papers  (2022-03-11T12:29:22Z)
• Fast End-to-End Speech Recognition via a Non-Autoregressive Model and Cross-Modal Knowledge Transferring from BERT [72.93855288283059]
  We propose a non-autoregressive speech recognition model called LASO (Listen Attentively, and Spell Once) The model consists of an encoder, a decoder, and a position dependent summarizer (PDS)
  arXiv  Detail & Related papers  (2021-02-15T15:18:59Z)
• CycAs: Self-supervised Cycle Association for Learning Re-identifiable Descriptions [61.724894233252414]
  This paper proposes a self-supervised learning method for the person re-identification (re-ID) problem. Existing unsupervised methods usually rely on pseudo labels, such as those from video tracklets or clustering. We introduce a different unsupervised method that allows us to learn pedestrian embeddings from raw videos, without resorting to pseudo labels.
  arXiv  Detail & Related papers  (2020-07-15T09:52:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.