The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs

• URL: http://arxiv.org/abs/2605.08737v1
• Date: Sat, 09 May 2026 06:48:00 GMT
• Title: The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs
• Abstract summary: ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
• Score: 52.709361620508595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy distillation (OPD) is widely used for LLM post-training. When pushed with a reward-extrapolation coefficient lambda > 1, the student can lift past the teacher in domain, but past a threshold lambda* the same step violates the output contract on structured-output tasks. In a single-position Bernoulli reduction, we derive a closed-form base-relative clip-safety threshold lambda*(p,b,c) determined by three measurable quantities: the teacher modal probability, the warm-start mass, and the importance-sampling clip strength. Above lambda*, the extrapolated fixed point exits the clip-safe region, changing training from format-preserving to format-collapsing. We extend the rule to calibrated K-ary listwise JSON tasks where a single binding equivalence class dominates the output contract and SFT retains parse headroom. On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval, a budget-extension test, and a small-clip cross-prediction--fall within their locked prediction windows, with the small-clip value matching the closed-form prediction below grid resolution. Operating just below lambda*, ListOPD brings a 1.7B Qwen3 student to in-domain parity with an 8B-SFT baseline at one-fifth the parameters. The gain is driven primarily by format adherence: NDCG@1 on parsed outputs remains flat across lambda, while parse validity sharply changes at the predicted boundary. The cliff diagnostic is rubric-independent, whereas the parity claim uses a Gemini-graded rubric and inherits that evaluator's exposure.

Related papers

• Constrained Online Learning with Noisy Constraint Values [55.29259818039367]
  Under common feasibility, our LEDGER algorithm attains $O(sqrt T)$ expected regret and $O(sqrtTlog(eT))$ expected budget violation.<n>Neither a Slater condition, independence between feedback channels, nor an absolute constraint-value bound is needed.
  arXiv  Detail & Related papers  (2026-09-07T01:38:41Z)
• Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and a Tight Univariate Rate [47.36630149538994]
  In high-dimensional online prediction, the best predictor may depend on only a few features, so regret should scale with sparsity rather than the ambient dimension.<n>Warmuth and Amid asked at COLT 2023 whether any of three such rules admits a competitive online regret guarantee.<n>Our analysis identifies a common obstruction: cheap nuisance causes the refit to underweight the truly predictive coordinate.
  arXiv  Detail & Related papers  (2026-08-18T09:32:03Z)
• Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification [1.5977080543655726]
  Common approaches such as verbalization produce extremely sparse outputs.<n>Qwen3-32B verbalizes only eight unique confidence values on SST-2, with over half being exactly 95%.<n>We address sparsity and the best AUARC (+2.3 points over vanilla verbalization) without incurring additional inference cost.
  arXiv  Detail & Related papers  (2026-08-05T14:24:34Z)
• CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning [4.663685189987781]
  Self-distilled agentic reinforcement learning augments trajectory-level reward with a token-level distillation loss.<n>The prevailing recipe gates this loss by a single scalar, the teacher-student log-probability gap.<n>We introduce CRAFT, a threepillar credit-assignment scheme that addresses both limitations.
  arXiv  Detail & Related papers  (2026-06-28T16:11:47Z)
• Filtered ANN as a Phase Transition: When Selectivity-Estimation Error Causes Plan Regret [0.0]
  We show that selectivity-estimation error produces plan regret -- recall lost versus the oracle strategy -- only in the critical regions around those boundaries.<n>A real approximate index does not mis-locate the boundary, but a biased cost model opens a persistent miscalibration band that estimation-error robustness cannot fix.
  arXiv  Detail & Related papers  (2026-06-15T07:44:58Z)
• DISA: Offline Importance Sampling for Distribution-Matching LLM-RL [56.9445657766829]
  We introduce DISA, which moves this calibration problem outside the RL loop.<n> DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate.<n>On two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline Flow.
  arXiv  Detail & Related papers  (2026-05-17T07:14:44Z)
• KL for a KL: On-Policy Distillation with Control Variate Baseline [13.281263788199219]
  On-Policy Distillation (OPD) has emerged as a dominant post-training paradigm for large language models.<n>We propose vOPD, which casts OPD as policy-gradient RL.<n>We show that vOPD consistently outperforms vanilla OPD and matches the most expensive full-vocabulary baseline.
  arXiv  Detail & Related papers  (2026-05-08T15:24:51Z)
• Two Calls, Two Moments, and the Vote-Accuracy Curve of Repeated LLM Inference [4.28787537081191]
  We study the binary correctness layer of repeated LLM inference under conditional-i.i.d. calls.<n>Every fixed majority-vote budget has a sharp distribution-free two-call interval.<n>Experiments on LLM calls over QNLI and QQP show that empirical three- and five-vote accuracies are contained in the projected two-call regions.
  arXiv  Detail & Related papers  (2026-05-05T05:40:09Z)
• When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning [60.714416943748866]
  In single-stream autoregressive interfaces, the same tokens both update the model state and constitute an irreversible public commitment.<n>We introduce Side-by-Side (SxS) Interleaved Reasoning, which makes disclosure timing a controllable decision within standard autoregressive generation.
  arXiv  Detail & Related papers  (2026-05-05T02:59:58Z)
• Posterior Augmented Flow Matching [64.1559809786948]
  Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
  arXiv  Detail & Related papers  (2026-05-01T17:59:59Z)
• PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners [9.324642081509756]
  Large language model (LLM) reasoning requires supervision that is both aligned with the model's own test-time states and informative at the token level.<n>Recent privileged on-policy self-distillation explores a middle ground by scoring student rollouts with the same model under verified solution context.<n>We propose PAINT, which masks the verified solution according to rollout-reference overlap and applies a small energy-space on a sparse set of entropy-mismatch token positions.
  arXiv  Detail & Related papers  (2026-04-29T11:56:07Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• Conformal Tradeoffs: Guarantees Beyond Coverage [0.02648566468224904]
  Deployed conformal predictors are long-lived decision infrastructure operating over finite operational windows.<n> Marginal coverage does not determine deployment-facing quantities.<n>We provide a framework for operational certification and planning beyond coverage.
  arXiv  Detail & Related papers  (2026-02-20T07:58:25Z)
• Ensemble Threshold Calibration for Stable Sensitivity Control [0.0]
  We present an end-to-end framework that achieves exact recall with sub-percent variance over tens of millions of geometry pairs.<n>Our approach consistently hits a recall target within a small error, decreases redundant verifications relative to other calibrations, and runs end-to-end on a single TPU v3 core.
  arXiv  Detail & Related papers  (2025-10-02T15:22:28Z)
• On the Practicality of Differential Privacy in Federated Learning by Tuning Iteration Times [51.61278695776151]
  Federated Learning (FL) is well known for its privacy protection when training machine learning models among distributed clients collaboratively. Recent studies have pointed out that the naive FL is susceptible to gradient leakage attacks. Differential Privacy (DP) emerges as a promising countermeasure to defend against gradient leakage attacks.
  arXiv  Detail & Related papers  (2021-01-11T19:43:12Z)
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient [62.24615324523435]
  This paper provides a statistical analysis of high-dimensional batch Reinforcement Learning (RL) using sparse linear function approximation. When there is a large number of candidate features, our result sheds light on the fact that sparsity-aware methods can make batch RL more sample efficient.
  arXiv  Detail & Related papers  (2020-11-08T16:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.