PROOF-Gen: From Optimized Data to Better Distillation

• URL: http://arxiv.org/abs/2608.23911v1
• Date: Mon, 24 Aug 2026 23:33:56 GMT
• Title: PROOF-Gen: From Optimized Data to Better Distillation
• Abstract summary: Post-training pipelines that drive tool-calling agents re-run this stage on a daily or weekly cadence, paying the frontier-teacher cost each cycle.<n>On 2-bench, 57% of teacher trials fail, two-thirds of them near-misses.<n>We introduce PROOF-Gen, which recovers golden trajectories from these failures via per-scenario prompt optimization.
• Score: 6.158612515104146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Supervised fine-tuning on teacher-generated trajectories is the standard first stage for distilling tool-calling capabilities into deployable models. Post-training pipelines that drive shipped tool-calling agents re-run this stage on a daily or weekly cadence, paying the frontier-teacher cost each cycle, yet the mechanism is generate-and-filter (keep the teacher's passing trajectories, discard the rest) and each cycle leaves behind the same hard scenarios because failures supply no signal. On τ2-bench, 57% of teacher trials fail, two-thirds of them near-misses (most tool calls correct, undone by one decisive error). We introduce PROOF-Gen (Per-scenario Reflective Optimization to Overcome FailedGeneration), which recovers golden trajectories from these failures via per-scenario prompt optimization. For each failed task, a reflector analyzes the execution trace and evaluation feedback, then writes corrective guidance that steers the teacher to a passing trajectory. The guidance is stripped before training, so the student learns from clean demonstrations with no task-specific scaffold. On τ2-bench, per-scenario optimization recovers 93% of failed scenarios. Fine-tuned on the combined data, Qwen3-4B-Instruct-2507 improves from Pass^1=0.132 to 0.529 and Gemma 4 E4B-it gains +7.2pp on BFCL v4 multi-turn. In a deployed pipeline, the method lifts trajectory quality by +6.3pp goal completion and transfers to a deployed on-device model (+1.5pp goal completion; +1.7 to +5.0pp across response-quality metrics), with positive transfer in every locale (non-English average +1.48pp).

Related papers

• ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training [66.72708893922605]
  Post-training of large multimodal models should add new capabilities while preserving those from pre-training.<n>SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting.<n>We introduce ReDraft, which obtains both from the model's own failures.
  arXiv  Detail & Related papers  (2026-09-15T04:59:03Z)
• T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks [27.750923745153145]
  T1 is a Mixture-of-Experts model of 122B total trained with reinforcement learning.<n>It operates a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier.<n>On Terminal-Bench 2.1, our post-train pipeline raises initial base model from 43.8% to T1 with 64.0% resolved.
  arXiv  Detail & Related papers  (2026-09-10T03:42:01Z)
• NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness [76.6324710887112]
  NeoHorse-1 is a family of agent-native models developed to explore this path through agentic post-training.<n>Our system combines a heterogeneous model pool with intelligent routing, recording the predicted capability demand, selected service tier, and subsequent interaction for each user turn.
  arXiv  Detail & Related papers  (2026-09-08T03:14:54Z)
• On-Policy Self-Distillation in Diffusion Models [89.32656931795293]
  Reinforcement learning can align diffusion models with human preferences and task-specific objectives, but endpoint rewards do not specify how an intermediate denoising prediction should change.<n>We introduce DiffusionOPSD as an on-policy self-distillation framework that converts image-level reward guidance into explicit targets for clean-output predictions at sampled queries.<n>Across SD 3.5-M and the step-distilled Z-Image-Turbo, our approach achieves the best final held-out scores in 19 of 20 reward-matched settings across two backbones and ten evaluators.
  arXiv  Detail & Related papers  (2026-08-25T14:55:24Z)
• Write, Execute, Refine: From Skill Followers to Skill Optimizers via Reinforcement Learning from Execution Feedback [52.78692582507693]
  Expert-written natural language skills can improve tool-using agents, yet agent-authored skills perform 8-11 points worse than using no skill.<n>We study how to organize execution experience from intermediate skills into training states for an executor.
  arXiv  Detail & Related papers  (2026-08-18T09:52:48Z)
• Cross-Benchmark Generalization in Long-Horizon Agents [1.0742728354283815]
  We post-train an open-weight mixture-of-experts model on 363 long-horizon Model Context Protocol (MCP) tasks across 27 categories.<n>No externalbenchmark task or grader entered training and no external score informed the reward, training hyper parameters, trained-checkpoint selection, or stopping.<n>Both software benchmarks improve despite the training collection containing no software-engineering tasks.
  arXiv  Detail & Related papers  (2026-07-31T18:05:36Z)
• Pass the Baton: Trajectory-Relayed On-Policy Distillation [23.50689651536186]
  On-policy distillation grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure.<n>We identify a teacher-student continuation asymmetry on failed prefixes, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD)<n>During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory.
  arXiv  Detail & Related papers  (2026-07-28T17:59:46Z)
• Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 [53.71882250666667]
  Most reported gains from agent-optimization methods are one-shot.<n>This does not test the setting that matters for deployed agents.<n>We compare three approaches to agent-harness optimization.
  arXiv  Detail & Related papers  (2026-07-15T16:36:04Z)
• Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1 [4.486629113012585]
  Recent progress on ARC-AGI-1 has come from two regimes: heavy test-time compute over frontier models, or benchmark-specific training in which small models are fine-tuned on ARC data.<n>We study what is recoverable through architecture alone, building agentic harnesses that decompose pattern-discovery and program-synthesis stages explicitly.
  arXiv  Detail & Related papers  (2026-07-07T19:49:35Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents [56.31499185764872]
  Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents.<n>We propose Patches-to-Trajectories (P2T), which uses $pstar$ as privileged information during curation and formulates trajectory construction as bi-objective optimization over per-step effectiveness and trajectory length.
  arXiv  Detail & Related papers  (2026-05-21T04:54:55Z)
• Rethinking Supervision Granularity: Segment-Level Learning for LLM-Based Theorem Proving [8.420305724549603]
  Automated theorem proving with large language models in Lean 4 is commonly approached through either step-level tactic prediction with tree search or whole-proof generation.<n>These two paradigms represent opposite granularities for constructing supervised training data.<n>We propose segment-level supervision, a training data construction strategy that extracts locally coherent proof segments for training policy models.
  arXiv  Detail & Related papers  (2026-05-12T10:18:28Z)
• Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation [86.62036852878354]
  Large language models (LLMs) increasingly operate in settings that require reliable long-context understanding.<n>We propose RoPE-Perturbed Self-Distillation, a training regularizer that improves positional robustness.<n> Experiments on long-context adaptation of Llama-3-8B and Qwen-3-4B demonstrate consistent gains on long-context benchmarks.
  arXiv  Detail & Related papers  (2026-04-15T18:46:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.