Related papers: $π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data

$π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data

URL: http://arxiv.org/abs/2604.14054v1
Date: Wed, 15 Apr 2026 16:34:39 GMT
Title: $π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data
Abstract summary: We propose Privileged Information Self-Play ($$-Play), a multi-agent self-evolution framework.<n>In $$-Play, an examiner generates tasks together with their QCPs, and a teacher model leverages QCP as privileged context to densely supervise a student via self-distillation.<n>Experiments show that data-free $$-Play surpasses fully supervised search agents and improves evolutionary efficiency by 2-3$times$ over conventional self-play.
Score: 57.79227052653548
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Deep search agents have emerged as a promising paradigm for addressing complex information-seeking tasks, but their training remains challenging due to sparse rewards, weak credit assignment, and limited labeled data. Self-play offers a scalable route to reduce data dependence, but conventional self-play optimizes students only through sparse outcome rewards, leading to low learning efficiency. In this work, we observe that self-play naturally produces a question construction path (QCP) during task generation, an intermediate artifact that captures the reverse solution process. This reveals a new source of privileged information for self-distillation: self-play can itself provide high-quality privileged context for the teacher model in a low-cost and scalable manner, without relying on human feedback or curated privileged information. Leveraging this insight, we propose Privileged Information Self-Play ($π$-Play), a multi-agent self-evolution framework. In $π$-Play, an examiner generates tasks together with their QCPs, and a teacher model leverages QCP as privileged context to densely supervise a student via self-distillation. This design transforms conventional sparse-reward self-play into a dense-feedback self-evolution loop. Extensive experiments show that data-free $π$-Play surpasses fully supervised search agents and improves evolutionary efficiency by 2-3$\times$ over conventional self-play.

Related papers

Latent On-Policy Self-Distillation [55.57800223145407]
On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories.<n>Existing methods still rely heavily on designer-specified privileged artifacts.<n>We introduce Latent On-Policy Self-Distillation (LOPD), which makes the teacher's privileged context itself learnable end-to-end from experience.
arXiv Detail & Related papers (2026-08-13T10:05:51Z)
Self-evolving LLM agents with in-distribution Optimization [60.05867547965365]
Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments.<n>We propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning.<n>We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance.
arXiv Detail & Related papers (2026-06-05T15:09:52Z)
Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text [14.278605706996474]
Self-play has emerged as a promising paradigm to train Large Language Models (LLMs)<n>We propose POP, a self-play framework that uses the same LLM to synthesize evaluation rubrics, along with input-output pairs, for each example.<n>On Qwen-2.5-7B, POP increases performance of both pretrained and instruction-tuned models, across different tasks.
arXiv Detail & Related papers (2026-04-21T23:21:56Z)
SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval [61.73547199231695]
SubSearch is a specialized framework that shifts from outcome-only supervision to intermediate reward signals.<n> Experiments show that rewarding intermediate reasoning steps with intrinsic rewards leads to more robust reasoning traces.<n>SubSearch can help in building reasoning traces that allow agents to better integrate search engines for complex query answering.
arXiv Detail & Related papers (2026-04-08T13:09:47Z)
Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain [22.77669491242655]
Large language models (LLMs) make it plausible to build systems that improve through self-evolving loops.<n>We show that sustainable self-evolution requires a self-synthesised data pipeline with learnable information that increases across iterations.
arXiv Detail & Related papers (2026-02-10T08:12:09Z)
Dr. Zero: Self-Evolving Search Agents without Training Data [34.91191770652202]
We introduce Dr. Zero, a framework enabling search agents to effectively self-evolve without any training data.<n>In particular, we design a self-evolution feedback loop where a proposer generates diverse questions to train a solver from the same base model.<n>To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO)
arXiv Detail & Related papers (2026-01-11T20:27:55Z)
AgentEvolver: Towards Efficient Self-Evolving Agent System [51.54882384204726]
We present AgentEvolver, a self-evolving agent system that drives autonomous agent learning.<n>AgentEvolver introduces three synergistic mechanisms: self-questioning, self-navigating, and self-attributing.<n>Preliminary experiments indicate that AgentEvolver achieves more efficient exploration, better sample utilization, and faster adaptation compared to traditional RL-based baselines.
arXiv Detail & Related papers (2025-11-13T15:14:47Z)
Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection [71.8243083897721]
Vision-language models often hallucinate details, generating non-existent objects or inaccurate attributes that compromise output reliability.<n>We present a novel framework that leverages the model's self-consistency between long responses and short answers to generate preference pairs for training.
arXiv Detail & Related papers (2025-09-27T10:37:11Z)
Can Large Reasoning Models Self-Train? [51.0277533541394]
We use majority voting as a simple self-feedback mechanism to study whether self-training can be sustained within reinforcement learning.<n>We find that this basic approach improves not only the model's reasoning performance, but also its capability of generating better quality feedback for the next RL iteration.<n>Yet our analysis also reveals a critical limitation of such a self-training paradigm - prolonged RL with self-reward leads to reward hacking, resulting in sudden and complete performance collapse.
arXiv Detail & Related papers (2025-05-27T17:16:00Z)
PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning [102.36450942613091]
We propose an inverse reinforcement learning algorithm, called emphinverse temporal difference learning (ITD) We show how to seamlessly integrate ITD with learning from online environment interactions, arriving at a novel algorithm for reinforcement learning with demonstrations, called $Psi Phi$-learning.
arXiv Detail & Related papers (2021-02-24T21:12:09Z)
Cost-effective Variational Active Entity Resolution [4.238343046459798]
We devise an entity resolution method that builds on the robustness conferred by deep autoencoders to reduce human-involvement costs. Specifically, we reduce the cost of training deep entity resolution models by performing unsupervised representation learning. Finally, we reduce the cost of labelling training data through an active learning approach that builds on the properties conferred by the use of deep autoencoders.
arXiv Detail & Related papers (2020-11-20T13:47:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.