Related papers: Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis

Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis

URL: http://arxiv.org/abs/2605.14392v1
Date: Thu, 14 May 2026 05:14:45 GMT
Title: Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis
Abstract summary: We pursue a vision for self-improving language models in which the model does not merely generate problems or traces to imitate, but constructs the environments that train it.<n>We instantiate this view in EvoEnv, a single-policy generator, solver method that synthesizes Python environments from ten seeds.
Score: 37.03537014865641
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Abstract: We pursue a vision for self-improving language models in which the model does not merely generate problems or traces to imitate, but constructs the environments that train it. In zero-data reasoning RL, this reframes self-improvement from a data-generation loop into an environment-construction loop, where each artifact is a reusable executable object that samples instances, computes references, and scores responses. Whether this vision sustains improvement hinges on a single property: the environments must exhibit stable solve--verify asymmetry, the model must be able to write an oracle once that it cannot reliably execute in natural language on fresh instances. This asymmetry takes two complementary forms. Some tasks are algorithmically hard to reason through but trivial as code: a dynamic program or graph traversal, compiled once, yields unboundedly many calibrated instances. Others are intrinsically hard to solve but easy to verify, like planted subset-sum or constraint satisfaction. Both create a durable gap between proposing and solving that the policy cannot close by gaming the verifier, and it is this gap that keeps reward informative as the learner improves. We instantiate this view in EvoEnv, a single-policy generator, solver method that synthesizes Python environments from ten seeds and admits them only after staged validation, semantic self-review, solver-relative difficulty calibration, and novelty checks. The strongest evidence comes from the already-strong regime: on Qwen3-4B-Thinking, fixed public-data RLVR and fixed hand-crafted environment RLVR reduce the average, while EvoEnv improves it from 72.4 to 74.8, a relative gain of 3.3%. Stable self-improvement, we suggest, depends not on producing more synthetic data, but on models learning to construct worlds whose difficulty stays structurally beyond their own reach.

Related papers

SPADE: Self-Play in Adaptive Synthetic Executable Environments [110.80471334001574]
We introduce SPADE, a self-play RL framework in which a single LLM plays two roles.<n>The Environment Designer writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface.<n>We find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory.
arXiv Detail & Related papers (2026-08-19T17:58:56Z)
Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoning [13.767813688729161]
Existing approaches largely rely on large-scale, statically curated datasets.<n>Ouroboros-Spatial is a self-evolving training framework in which the model plays dual roles as a proposer and a solver.<n>Ouroboros-Spatial substantially improves Qwen3-VL-4B and Qwen3-VL-8B while using an order of fewer training examples than recent large-scale curated datasets.
arXiv Detail & Related papers (2026-06-10T06:49:21Z)
Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline [56.53954182896384]
We propose Self-Verified Distillation, a simple post-training refinement algorithm for large language models.<n>Self-Verified Distillation generates candidate solutions to unlabeled seed questions.<n>It filters them using prompt-based self-verification, and trains on the resulting self-curated dataset.<n>We find that sampling more candidate generations and using a larger verification budget during training data construction produces higher-quality self-curated data.
arXiv Detail & Related papers (2026-05-20T17:26:10Z)
EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation [16.36266898493489]
A promising direction is to replace manually crafted environments with LLM-simulated counterparts.<n>This paradigm hinges on an unexamined core assumption: LLMs can accurately simulate environmental feedback.<n>In practice, LLM-simulated environments suffer from hallucinations, logical inconsistencies, and silent state drift failures.
arXiv Detail & Related papers (2026-05-08T05:08:16Z)
ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models [18.359969463106644]
Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising approach for training reasoning language models (RLMs)<n>In this work, we scale RLVR by introducing ReSyn, a pipeline that generates diverse reasoning environments equipped with instance generators and verifiers.
arXiv Detail & Related papers (2026-02-23T18:34:29Z)
DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution [38.660154251426505]
We introduce DARC (Decoupled Asymmetric Reasoning Curriculum), a two-stage framework that stabilizes the self-evolution process.<n>We show that DARC is model-agnostic, yielding an average improvement of 10.9 points across nine reasoning benchmarks and three backbone models.
arXiv Detail & Related papers (2026-01-20T09:12:27Z)
SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models [35.53535406831892]
We introduce a novel self-play fine-tuning method, namely Self-PlAy via Noise Contrastive Estimation (SPACE)<n>SPACE treats synthetic samples as auxiliary components, and discriminates them from the real ones in a binary classification manner.<n>We show that SPACE significantly improves the performance of LLMs over various tasks, and outperforms supervised fine-tuning that employs much more real-world samples.
arXiv Detail & Related papers (2025-12-08T05:16:18Z)
Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation [69.62857948698436]
Recent advances in large language models (LLMs) have improved their performance on coding benchmarks.<n>However, improvement is plateauing due to the exhaustion of readily available high-quality data.<n>We propose Sol-Ver, a self-play solver-verifier framework that jointly improves a single model's code and test generation capacity.
arXiv Detail & Related papers (2025-02-20T18:32:19Z)
Generative Modeling of Regular and Irregular Time Series Data via Koopman VAEs [50.25683648762602]
We introduce Koopman VAE, a new generative framework that is based on a novel design for the model prior. Inspired by Koopman theory, we represent the latent conditional prior dynamics using a linear map. KoVAE outperforms state-of-the-art GAN and VAE methods across several challenging synthetic and real-world time series generation benchmarks.
arXiv Detail & Related papers (2023-10-04T07:14:43Z)
Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding [76.89426311082927]
Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference. We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
arXiv Detail & Related papers (2021-04-13T17:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.