SPADE: Self-Play in Adaptive Synthetic Executable Environments

• URL: http://arxiv.org/abs/2608.19197v2
• Date: Mon, 24 Aug 2026 17:59:18 GMT
• Title: SPADE: Self-Play in Adaptive Synthetic Executable Environments
• Abstract summary: We introduce SPADE, a self-play RL framework in which a single LLM plays two roles.<n>The Environment Designer writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface.<n>We find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory.
• Score: 110.80471334001574
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.

Related papers

• EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL [54.09410318521061]
  We introduce EnvFactory, a fully automated framework for Agentic Reinforcement Learning (Agentic RL) training.<n>EnvFactory autonomously explores and stateful, executable tool environments from authentic resources.<n>It synthesizes natural multi-turn trajectories through topology-aware sampling and calibrated refinement.<n>It achieves superior training efficiency and downstream performance, improving Qwen3-series models by up to +15% on BFCLv3, +8.6% on MCP-Atlas, and +6% on conversational benchmarks including $2$-Bench and VitaBench.
  arXiv  Detail & Related papers  (2026-05-18T17:37:40Z)
• SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning [21.4605333924798]
  SimWorld Studio is an open-source platform built on Unreal Engine 5 for generating evolving embodied learning environments.<n>At its core is SimCoder, a tool/skill-augmented coding agent that writes and executes engine-level code to construct physically grounded 3D worlds.<n> generated worlds are exported as Gym-style environments for embodied agent learning.
  arXiv  Detail & Related papers  (2026-05-10T08:51:50Z)
• EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation [16.36266898493489]
  A promising direction is to replace manually crafted environments with LLM-simulated counterparts.<n>This paradigm hinges on an unexamined core assumption: LLMs can accurately simulate environmental feedback.<n>In practice, LLM-simulated environments suffer from hallucinations, logical inconsistencies, and silent state drift failures.
  arXiv  Detail & Related papers  (2026-05-08T05:08:16Z)
• TRACE: Capability-Targeted Agentic Training [11.088296897059157]
  We introduce TRACE, an end-to-end system for environment-specific agent self-improvement.<n>TRACE contrasts successful and failed trajectories to identify lacking capabilities.<n>It synthesizes a targeted training environment for each that rewards whether the capability was exercised.<n>It trains a LoRA adapter via RL on each synthetic environment, routing to the relevant adapter at inference.
  arXiv  Detail & Related papers  (2026-04-07T02:22:44Z)
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning [62.499592503950026]
  Large language model (LLM) have empowered autonomous agents to perform complex tasks that require multi-turn interactions with tools and environments.<n>We propose Agent World Model (AWM), a fully synthetic environment generation pipeline.<n>We scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets.
  arXiv  Detail & Related papers  (2026-02-10T18:55:41Z)
• EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis [101.67583081810136]
  Large language models (LLMs) are expected to be trained to act as agents in various real-world environments.<n>This process relies on rich and varied tool-interaction sandboxes.<n>We propose EnvScaler, an automated framework for scalable tool-interaction environments.
  arXiv  Detail & Related papers  (2026-01-09T14:32:06Z)
• Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem [90.17610617854247]
  We introduce the Agentic Learning Ecosystem (ALE), a foundational infrastructure that optimize the production pipeline for agentic model.<n>ALE consists of three components: ROLL, a post-training framework for weight optimization; ROCK, a sandbox environment manager for trajectory generation; and iFlow CLI, an agent framework for efficient context engineering.<n>We release ROME, an open-source agent grounded by ALE and trained on over one million trajectories.
  arXiv  Detail & Related papers  (2025-12-31T14:03:39Z)
• Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting [92.57796055887995]
  We introduce ECHO, a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents.<n> ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts.<n>We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation.
  arXiv  Detail & Related papers  (2025-10-11T18:11:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.