CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation

• URL: http://arxiv.org/abs/2607.16955v1
• Date: Sat, 18 Jul 2026 20:16:26 GMT
• Title: CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation
• Abstract summary: CADENCE is a unified framework with a targeted fix for each failure mode.<n>DRIFT schedules a per-token convex mixture of forward-KL and reverse-KL surrogate objectives on student-sampled trajectories.<n>Six components extend it: (A) COVA, a coverage-adaptive schedule accelerating the forward-to-reverse transition; (B) FTB, a forking-token boost concentrating gradient at high-entropy positions via a globally-normalized entropy reference; (D) LAP, brevity-preferential correct-rollout reinforcement; (E) EMR, an entropy-gnostic calibration
• Score: 2.587194279527956
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy knowledge distillation transfers reasoning from large teachers to compact students, but existing approaches suffer three compounding failure modes: (i) cold-start collapse, where a fresh student assigns near-zero mass to teacher-preferred tokens; (ii) state-agnostic divergence scheduling, where time-only forward/reverse-KL interpolation ignores the student's coverage state; and (iii) binary reward sparsity, where pass/fail signals discard information from partially correct traces. We present CADENCE, a unified framework with a targeted fix for each. Its DRIFT mechanism schedules a per-token convex mixture of forward-KL and reverse-KL surrogate objectives on student-sampled trajectories (per-token surrogates, not sequence-level KL gradient estimators). Six components extend it: (A) COVA, a coverage-adaptive $β$ schedule accelerating the forward-to-reverse transition; (B) FTB, a forking-token boost concentrating gradient at high-entropy positions via a globally-normalized entropy reference; (C) CCD, a dense reward adding numerical-proximity partial credit for incorrect-but-close traces; (D) LAP, brevity-preferential correct-rollout reinforcement; (E) EMR, an entropy-matching calibration regularizer; (F) BSD, a bootstrapped self-distillation phase. On GSM8K and MATH-500 (corrected 512-token protocol, 5 seeds, reported std), CADENCE distills a 0.5B student from a 1.5B teacher to 69.8 $\pm$ 0.5% GSM8K pass@1 (from 48.7% pretrained; 63.2% of the teacher gap closed) and to 72.1 $\pm$ 0.4% with a 3B teacher (76.2% closed), beating the strongest matched-compute label-using baseline (DRIFT+binary reward) by +4.4 $\pm$ 0.7 points. All experiments run on a single Apple Mac Studio (M-series, 64GB unified memory), showing principled distillation reaches strong reasoning quality without datacenter-scale hardware.

Related papers

• Bounded Adjustment with Reliability-Guided Embedding for Imbalanced Learning with Noisy Labels [7.047239027940408]
  We introduce BARGE (Bounded Adjustment with Reliability-Guided Embeddings), a single-stage objective combining a bounded, prior-adjusted density-power score with reliability-guided angular geometry.<n>We evaluate it on CIFAR-10, CIFAR-100, and Tiny ImageNet under long-tail and step imbalance, clean labels, and 20% and 40% random incorrect-label replacement.
  arXiv  Detail & Related papers  (2026-09-14T21:41:28Z)
• Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment [5.864346468963738]
  Multi-turn agentic RL increasingly treats credit assignment as a targeting problem.<n>We show that terminal-state verifiers sit deep in a low-V_d regime where targeting is the wrong axis.
  arXiv  Detail & Related papers  (2026-09-02T10:37:12Z)
• WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training [48.290951447286744]
  WDL-OPD is a mixture-constrained co-training method with two trainable policies.<n>An anchor policy generates every rollout, an auxiliary policy evaluates the same visited states, and a geometric mixture of their token distributions is matched to a frozen teacher by reverse KL.<n>We show that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD$2$ and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express.
  arXiv  Detail & Related papers  (2026-08-10T11:22:53Z)
• GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning [61.67411833252235]
  We introduce GradCuit, which inserts optimizable latent states at a selected Transformer layer between the hidden representations of the prompt and the generated continuation.<n>Across five instruction-tuned backbones, three reasoning benchmarks, and two answer formats, GradCuit achieves an average accuracy of 64.5%.<n>For interpretability, token-level gradient attribution reveals that latent influence concentrates on reasoning-connector tokens, while layer analysis identifies early-to-middle Transformer layers as the most effective optimization space.
  arXiv  Detail & Related papers  (2026-08-03T17:55:24Z)
• Discrete Action Space as a Prerequisite for GRPO Convergence in Small-Model Continuous Control [0.0]
  Group Relative Policy Optimization can fine-tune small language models for simulated quadrotor continuous-control tasks.<n>In our benchmark, vanilla GRPO fine-tuning of Qwen-0.5B for 25 Hz quadrotor velocity control collapses to the trivial zero action.<n>A high-fidelity simulation using Crazyflie 2.1 dynamics surfaces a hover-region training-distribution gap.
  arXiv  Detail & Related papers  (2026-07-07T17:14:19Z)
• CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning [4.663685189987781]
  Self-distilled agentic reinforcement learning augments trajectory-level reward with a token-level distillation loss.<n>The prevailing recipe gates this loss by a single scalar, the teacher-student log-probability gap.<n>We introduce CRAFT, a threepillar credit-assignment scheme that addresses both limitations.
  arXiv  Detail & Related papers  (2026-06-28T16:11:47Z)
• Deterministic Decisions for High-Stakes AI. A Zero-Egress Pipeline with the Deployability of RAG and the Accuracy of Machine Learning [0.0]
  Intervention bias is a previously unquantified failure mode of large-language-model educational advisory agents.<n>We show that supervised policy learning eliminates this bias.<n>We also show an Evaluation Gap: LLM-as-judge scoring is blind to intervention bias, rewarding fluent over-prescription rather than decision quality.
  arXiv  Detail & Related papers  (2026-06-28T08:58:17Z)
• PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation [17.919178151901573]
  We show that standard on-policy distillation (OPD) for large language models suffers from severe training pathologies in practice.<n>PowerOPD is a sign-consistent reward from the Box-Cox power transformation, parameterized by alpha > 0, of which the log-ratio is the degenerate alpha -> 0 limit.<n>It achieves benchmark-averaged Avg@8/Pass@8 gains of up to +6.37/+5.71 over vanilla OPD, +3.01/+3.54 over post-hoc stabilization, and +2.59/+8.90 over full-vocabulary OPD.
  arXiv  Detail & Related papers  (2026-06-15T18:37:51Z)
• CaricHarmony: Contrastive Diffusion Paths for Identity-Preserving Caricature Synthesis [49.596677723190886]
  Sketch-based caricature synthesis suffers from a fundamental failure mode.<n>Identity and shape conditions are combined in diffusion models, causing collapse toward bland portraits or unrecognizable distortions.<n>We present CaricHarmony, the first training-free method that explicitly resolves this contamination through parallel uncontaminated diffusion paths.
  arXiv  Detail & Related papers  (2026-06-11T22:57:59Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Non-Vacuous Certification of Transport MCMC via Oscillation-Controlled Normalizing Flows [3.8549131582427303]
  Transport MCMC trains a normalizing flow to precondition Metropolis--Hastings proposals.<n>We establish the first numerically non-vacuous, rigorous spectral-gap bound for such samplers.
  arXiv  Detail & Related papers  (2026-05-31T07:46:48Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents [56.31499185764872]
  Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents.<n>We propose Patches-to-Trajectories (P2T), which uses $pstar$ as privileged information during curation and formulates trajectory construction as bi-objective optimization over per-step effectiveness and trajectory length.
  arXiv  Detail & Related papers  (2026-05-21T04:54:55Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• SHIFT: Robust Double Machine Learning for Average Dose-Response Functions under Heavy-Tailed Contamination [0.0]
  SHIFT (Self-calibrated Heavy-tail Inlier-Fit with Tempering) is a robust DML estimator.<n>It combines cross-fit nuisanceization with a kernel-local Welsch-loss second stage optimized by Graduated Non-Convexity.<n>It has competitive worst-case shape recovery across 1,400 mainsweep fits.
  arXiv  Detail & Related papers  (2026-04-30T19:51:35Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.