Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning

• URL: http://arxiv.org/abs/2605.08741v1
• Date: Sat, 09 May 2026 07:06:37 GMT
• Title: Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning
• Abstract summary: In-time harnesses substantially improve large language models on complex reasoning tasks.<n>We introduce emphOn-Policy Harness Self-Distillation (OPHSD)<n>OPHSD internalizes task-specific harness capabilities into the student model.
• Score: 8.33682726719781
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inference-time harnesses substantially improve large language models on complex reasoning tasks. However, the intrinsic capabilities of the underlying model remain unchanged by the addition of these external workflows. To bridge this gap, we introduce \emph{On-Policy Harness Self-Distillation} (OPHSD), which employs the harness-augmented current model as a teacher for self-distillation, thereby introducing extra supervisory signals from the harness beyond training data. OPHSD internalizes task-specific harness capabilities into the student model, yielding robust generalizability and strong standalone performance across diverse reasoning tasks. Evaluated across draft--verify harness for text classification and plan--solve for mathematical reasoning tasks, OPHSD consistently outperforms strong baselines (e.g., +10.83\% over OPSD on HMMT25). Our analysis further indicates that reattaching the harness during inference yields no additional benefits and can even degrade performance, suggesting that complex harnesses need not always be permanent fixtures; instead, they can serve as temporary training scaffolds whose benefits are permanently fed back into the base model. Our code and training data are available at https://github.com/zzy1127/OPHSD-On-Policy-Harness-Self-Distillation.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• Harness Continual Learning: Continual Adaptation Beyond Model Parameters [9.54786133734043]
  We formulate a new continual learning paradigm in which the harness evolves around a frozen foundation model.<n>A candidate commits a harness only after checking current improvement, historical retention, and validity.<n>Experiments demonstrate capability accumulation and failure recovery, with relative gains exceeding 10% over corresponding baselines in multiple settings.
  arXiv  Detail & Related papers  (2026-08-19T15:12:31Z)
• Self-Evolving Embodied Agents via Skill-Harness Evolution [53.307052568223945]
  Embodied agents are increasingly built as systems around foundation models.<n>While supervised fine-tuning and reinforcement learning can adapt agents to new environments, they require additional data, rewards, and training runs.<n>We propose SHAPER, a self-evolving framework for train-free embodied adaptation that keeps model parameters frozen.
  arXiv  Detail & Related papers  (2026-08-11T18:55:58Z)
• Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents? [28.21326355616294]
  We instantiate Feedback-Augmented Self-Distillation (FA-SD), a self-distillation algorithm for agentic search.<n>We identify that models can rely on recurring reasoning-and-search output templates, producing trajectories that appear diverse but are largely agnostic to the input question.<n>We show that although the self-teacher achieves stronger performance, learning remains inherently unstable due to inconsistent supervision signals.
  arXiv  Detail & Related papers  (2026-07-20T05:07:22Z)
• Recursive Harness Self-Improvement [28.432258880153125]
  Under model--harness co-evolution, harnesses are data-generating components whose execution traces can shape future foundation models.<n>We introduce Recursive Harness Self-Improvement (RHI), which represents the harness as a prompt-level specification of the agent loop.<n>RHI iteratively refines it using pairwise feedback over its own revision history.
  arXiv  Detail & Related papers  (2026-07-17T00:21:19Z)
• Denser $\ eq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training [44.59915053122529]
  Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities.<n>Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach.<n>Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2026-07-02T06:24:30Z)
• Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents [82.27610290890475]
  We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving.<n>Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains.<n>Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier.
  arXiv  Detail & Related papers  (2026-05-28T22:16:14Z)
• SEIF: Self-Evolving Reinforcement Learning for Instruction Following [53.280277743734096]
  We propose SEIF (Self-Evolving Reinforcement Learning for Instruction Following), a self-evolving framework for enhancing the instruction-following ability of large language models (LLMs)<n>SEIF forms a closed self-evolution loop that improves the model's instruction-following ability, where instruction difficulty evolution and model capability evolution reinforce each other.<n> Experiments across multiple model scales and architectures show that SEIF consistently improves instruction-following performance, suggesting strong generality.
  arXiv  Detail & Related papers  (2026-05-08T09:13:12Z)
• UniSD: Towards a Unified Self-Distillation Framework for Large Language Models [38.12407816591177]
  Self-distillation offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers.<n>Existing methods mainly examine isolated design choices, leaving their effectiveness, roles, and interactions unclear.<n>We propose UniSD, a unified framework to systematically study self-distillation.
  arXiv  Detail & Related papers  (2026-05-07T17:22:11Z)
• LEGATO: Good Identity Unlearning Is Continuous [30.78585550012454]
  LEGATO is a machine unlearning method for learning to forget identity of generative models.<n>It achieves state-of-the-art forgetting performance, avoids catastrophic collapse and reduces fine-tuned parameters.<n>Our experiments show that LEGATO achieves state-of-the-art forgetting performance, avoids catastrophic collapse and reduces fine-tuned parameters.
  arXiv  Detail & Related papers  (2026-01-07T13:15:25Z)
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness [49.72591739116668]
  Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs)<n>However, prevalent methods like GRPO often fail when task difficulty exceeds the model's capacity, leading to reward sparsity and inefficient training.<n>We propose HINT: Helping Ineffective rollouts Navigate Towards effectiveness, an adaptive hinting framework.
  arXiv  Detail & Related papers  (2025-10-10T13:42:03Z)
• Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning [9.120944934920141]
  We study trainable rational activations in both reinforcement and continual learning settings.<n>Our main result is demonstrating a trade-off between expressivity and plasticity in rational activations.<n>Our findings provide actionable design principles for robust and trainable activations in dynamic, nonstationary environments.
  arXiv  Detail & Related papers  (2025-07-19T19:53:08Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening [51.34904967046097]
  Selective Synaptic Dampening (SSD) is a fast, performant, and does not require long-term storage of the training data. We present a novel two-step, post hoc, retrain-free approach to machine unlearning which is fast, performant, and does not require long-term storage of the training data.
  arXiv  Detail & Related papers  (2023-08-15T11:30:45Z)
• Improving GANs with A Dynamic Discriminator [106.54552336711997]
  We argue that a discriminator with an on-the-fly adjustment on its capacity can better accommodate such a time-varying task. A comprehensive empirical study confirms that the proposed training strategy, termed as DynamicD, improves the synthesis performance without incurring any additional cost or training objectives.
  arXiv  Detail & Related papers  (2022-09-20T17:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.