Diagnosing and Calibrating Tool-Call Boundary Drift in Multi-Teacher On-Policy Distillation

• URL: http://arxiv.org/abs/2607.07050v2
• Date: Wed, 15 Jul 2026 06:02:40 GMT
• Title: Diagnosing and Calibrating Tool-Call Boundary Drift in Multi-Teacher On-Policy Distillation
• Abstract summary: We show that multi-teacher on-policy distillation can induce a behavior shift invisible from aggregate losses alone.<n>In a two-teacher tool-use setting, vanilla generalized knowledge distillation improves tool-call recall but also moves the model toward over-calling.<n>We propose Soft Clamp, a per-token divergence calibration method that dynamically compresses extreme token-level Jensen-Shannon divergence.
• Score: 6.100373666222075
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agentic language models must learn when to call tools, when to consume tool responses, and when to answer directly. This makes multi-teacher on-policy distillation a natural training strategy: one teacher can specialize in tool calls, another in direct responses, and the student can learn from both on its own generated distribution. We show that this strategy can induce a behavior shift that is invisible from aggregate losses alone. In a two-teacher tool-use setting, vanilla generalized knowledge distillation improves tool-call recall but also moves the model toward over-calling, where it calls tools on examples that should be answered directly. Aggregate explanations are insufficient: tool-call samples do not receive more token exposure, and full-sequence per-token divergence is not larger for the tool-call teacher. We instead analyze behavior leverage imbalance: local token-level signals at mode- entry and structural positions, such as <tool_call> and function names, can have disproportionate control over the global generation mode. We propose Soft Clamp, a per-token divergence calibration method that dynamically compresses extreme token-level Jensen-Shannon divergence while preserving nonzero gradients. On APIGen-MT, Soft Clamp reduces over-calling from 13.7% to 9.0% relative to vanilla GKD while matching its decision accuracy. In a BFCL multi-turn diagnostic, it also lowers tool-call loops and repeated calls among GKD variants. These results suggest that multi-teacher OPD should monitor where teacher signals act, not only how large they are in aggregate.

Related papers

• Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling [65.03665015319457]
  We propose an action-class-oriented diagnostic framework that decomposes multi-turn failures into two modes: action-class miscalibration and action-execution failure.<n>We validate it on a panel of tool-calling models across multiple multi-turn benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T09:08:15Z)
• Tunable Tool-Call Rates in LLM Agents via Representation Steering [45.332672398938136]
  We show that a tool can be controlled by a single linear direction in its residual stream, extracted without any training from the model's own tool-use preference signal.<n>We also show that the direction generalizes to unseen tools with strength comparable to each tool's own direction without favoring any specific tool choice.
  arXiv  Detail & Related papers  (2026-08-25T22:35:18Z)
• Graph Is the Verifier: Agentic Reinforcement Learning for Interprocedural Vulnerability Detection [13.409515289348532]
  VulAgentRL is a framework for interprocedural vulnerability detection built on a Code Property Graph (CPG)<n>We show that VulAgentRL outperforms state-of-the-art baselines, including frontier models, on the strict pair-wise-correct metric while issuing fewer tool calls.
  arXiv  Detail & Related papers  (2026-07-29T09:16:54Z)
• Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It [23.538941671680806]
  Tool use enables large language models to perform complex tasks.<n>Recent agentic reinforcement learning methods show promise for enhancing model capabilities.<n>Some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail.
  arXiv  Detail & Related papers  (2026-06-24T16:55:56Z)
• OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification [21.095025092772257]
  On-Policy Distillation (OPD) trains a student model on its own generative trajectories under dense token-level feedback from a stronger teacher.<n>We introduce OmniOPD, a novel framework that addresses both limitations through a logit-free, chunk-level supervision signal.
  arXiv  Detail & Related papers  (2026-05-31T22:31:15Z)
• SOD: Step-wise On-policy Distillation for Small Language Model Agents [32.49707795291693]
  Tool-integrated reasoning is difficult to scale to small language models.<n>Recently, on-policy distillation has gained popularity by supplying dense token-level supervision from a teacher.<n>We propose SOD, a step-wise on-policy distillation framework for small language model agents.
  arXiv  Detail & Related papers  (2026-05-08T13:30:42Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors [66.18091962164219]
  Existing metrics fail to distinguish mandatory behaviors required for task success from non-mandatory patterns that reflect a model's autonomous preferences.<n>We propose two complementary metrics to isolate non-mandatory behavioral patterns: textbfResponse Pattern Similarity (RPS) for verbal alignment and textbfAction Graph Similarity (AGS) for tool-use habits modeled as directed graphs.
  arXiv  Detail & Related papers  (2026-04-23T03:48:56Z)
• Distillation Traps and Guards: A Calibration Knob for LLM Distillability [54.90137955363471]
  We propose a post-hoc calibration method that enables control over a teacher's distillability via reinforcement fine-tuning.<n>Our objective combines task utility, KL anchor, and across-tokenizer calibration reward.<n> Experiments show that students distilled from distillable teachers outperform SFT and KD baselines.
  arXiv  Detail & Related papers  (2026-04-21T01:22:35Z)
• Self-Distilled RLVR [57.37526213765131]
  We show that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training.<n>We propose textbfRLSD (textbfRLVR with textbfSelf-textbfDistillation)<n>This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.
  arXiv  Detail & Related papers  (2026-04-03T15:50:07Z)
• HSFM: Hard-Set-Guided Feature-Space Meta-Learning for Robust Classification under Spurious Correlations [25.503211313768062]
  We propose a bilevel meta-learning method that performs augmentation directly in feature space to improve spurious correlation handling in the classifier head.<n>By operating at the backbone output rather than in pixel space, the method is highly efficient and stable, requiring only a few minutes of training on a single GPU.
  arXiv  Detail & Related papers  (2026-03-31T06:32:56Z)
• Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes [31.95045602299568]
  On-policy distillation (OPD) is appealing for large language model (LLM) post-training because it evaluates teacher feedback on student-generated rollouts rather than fixed teacher traces.<n>We revisit OPD from the estimator and implementation sides.<n>We identify three failure modes of sampled-token OPD: an imbalanced one-token signal, unreliable teacher guidance on student-generated prefixes, and distortions caused by tokenizer or special-token mismatch.
  arXiv  Detail & Related papers  (2026-03-26T15:35:59Z)
• DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation [68.19756761027351]
  Diffusion large language models (dLLMs) are compelling alternatives to autoregressive (AR) models.<n>We investigate their denoising processes and reinforcement learning methods.<n>Our work provides deeper insight into the machinery of dLLM generation and offers an effective, diffusion-native RL training framework.
  arXiv  Detail & Related papers  (2025-06-25T17:35:47Z)
• Multi-Granularity Semantic Revision for Large Language Model Distillation [66.03746866578274]
  We propose a multi-granularity semantic revision method for LLM distillation. At the sequence level, we propose a sequence correction and re-generation strategy. At the token level, we design a distribution adaptive clipping Kullback-Leibler loss as the distillation objective function. At the span level, we leverage the span priors of a sequence to compute the probability correlations within spans, and constrain the teacher and student's probability correlations to be consistent.
  arXiv  Detail & Related papers  (2024-07-14T03:51:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.