PACE: Parameter Change for Unsupervised Environment Design

• URL: http://arxiv.org/abs/2605.01358v1
• Date: Sat, 02 May 2026 10:07:59 GMT
• Title: PACE: Parameter Change for Unsupervised Environment Design
• Abstract summary: Unsupervised Environment Design offers promising paradigm for improving reinforcement learning generalization by adaptively shaping training environments.<n>Existing UED methods evaluate environments using indirect proxy signals such as regret, value-based errors, or Monte Carlo, which suffer from bias, high variance, or substantial computational overhead.<n>We propose Change Environment Design (PACE), which evaluates an environment through the policy change induced by training on that environment, directly grounding environment selection in realized learning progress.
• Score: 22.066871262223312
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unsupervised Environment Design (UED) offers a promising paradigm for improving reinforcement learning generalization by adaptively shaping training environments, but it requires reliable environment evaluation to remain effective. However, existing UED methods evaluate environments using indirect proxy signals such as regret, value-based errors, or Monte Carlo, which suffer from bias, high variance, or substantial computational overhead and fail to reflect agent realized learning progress. To address these limitations, we propose Parameter Change Environment Design (PACE), which evaluates an environment through the policy parameter change induced by training on that environment, directly grounding environment selection in realized learning progress. Specifically, PACE assigns environment value using a first-order approximation of the policy optimization objective, where the improvement induced by an environment is proportional to the squared L2 norm of the corresponding parameter update, enabling low-variance and computation-efficient evaluation without additional rollouts. Experiments on MiniGrid and Craftax show that PACE consistently outperforms established UED baselines, achieving higher IQM and smaller Optimality Gap on OOD evaluations, including an IQM of 96.4% and an Optimality Gap of 17.2% on MiniGrid.

Related papers

• EnvHarness: Awakening Static Worlds for Agent Learning [58.54600450547412]
  Environment Harness wraps a static environment to reshape its behavior without modifying the underlying logic.<n>EnvHarness applies across diverse domains while ensuring every reshaped environment retains its original verifier.<n>EnvRigger treats the target policy as a black box, observing its execution trajectories to synthesize EnvHarness components targeting diagnosed flaws.
  arXiv  Detail & Related papers  (2026-08-20T10:42:06Z)
• Curriculum Generation under Structured Parametric Environments for Robust Navigation Policies [0.0]
  This paper proposes a curriculum generation framework for structured continuous environment parameters using unidirectional gradient-based optimization.<n>The proposed method is evaluated across two continuous-control OpenAI Gym environments.<n>Across five random seeds, our method consistently outperforms vanilla policy training, random parameter sampling, manual curricula, frontier-based methods, Self-Paced Reinforcement Learning (SPRL), Absolute Learning Progress with Gaussian Mixture Models (ALP-GMM), and reverse curriculum learning baselines.
  arXiv  Detail & Related papers  (2026-08-09T07:45:21Z)
• Actor-Critic with Active Importance Sampling [61.41272490044518]
  This paper introduces the Active-Importance-Sampling Actor-Critic (AISAC) algorithm.<n>AISAC is an extension of the Actor-Critic framework for reducing variance in policy gradient estimation.<n> Experiments show improved learning speed, sample efficiency, and training compared to standard Actor-Critic methods.
  arXiv  Detail & Related papers  (2026-05-08T01:21:32Z)
• ProAct: Agentic Lookahead in Interactive Environments [56.50613398808361]
  ProAct is a framework that enables agents to internalize accurate lookahead reasoning through a two-stage training paradigm.<n>We introduce Grounded LookAhead Distillation (GLAD), where the agent undergoes supervised fine-tuning on trajectories derived from environment-based search.<n>We also propose the Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator designed to enhance policy-gradient algorithms.
  arXiv  Detail & Related papers  (2026-02-05T05:45:16Z)
• Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models [52.48582333951919]
  We propose a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates.<n>SAGE (Stability-Aware Gradient Efficiency) integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines.
  arXiv  Detail & Related papers  (2026-02-01T12:56:10Z)
• Improving Regret Approximation for Unsupervised Dynamic Environment Generation [19.50608711043436]
  Unsupervised Environment Design (UED) seeks to automatically generate training curricula for reinforcement learning (RL) agents.<n>Current methods struggle with a difficult credit assignment problem and rely on regret approximations that fail to identify challenging levels.<n>We propose Dynamic Environment Generation for UED to enable a denser level generator reward signal.
  arXiv  Detail & Related papers  (2026-01-21T12:58:40Z)
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning [77.92320830700797]
  Reinforcement Learning has played a central role in enabling reasoning capabilities of Large Language Models.<n>We propose a tractable computational framework that tracks and leverages curvature information during policy updates.<n>The algorithm, Curvature-Aware Policy Optimization (CAPO), identifies samples that contribute to unstable updates and masks them out.
  arXiv  Detail & Related papers  (2025-10-01T12:29:32Z)
• Improving Environment Novelty Quantification for Effective Unsupervised Environment Design [7.973747521623636]
  Unsupervised Environment Design (UED) formalizes the problem of autocurricula through interactive training between a teacher agent and a student agent.<n>Existing UED methods mainly rely on regret, a metric that measures the difference between the agent's optimal and actual performance.<n>This paper introduces the Coverage-based Evaluation of Novelty In Environment (CENIE) framework.
  arXiv  Detail & Related papers  (2025-02-08T23:59:41Z)
• Certifiably Robust Policies for Uncertain Parametric Environments [57.2416302384766]
  We propose a framework based on parametric Markov decision processes (MDPs) with unknown distributions over parameters.<n>We learn and analyse IMDPs for a set of unknown sample environments induced by parameters.<n>We show that our approach produces tight bounds on a policy's performance with high confidence.
  arXiv  Detail & Related papers  (2024-08-06T10:48:15Z)
• Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification [22.241676350331968]
  This study focuses on scenarios involving a simulation environment with uncertainty parameters and the set of their possible values. The aim is to optimize the worst-case performance on the uncertainty parameter set to guarantee the performance in the corresponding real-world environment. Experiments in multi-joint dynamics with contact (MuJoCo) environments show that the proposed method exhibited a worst-case performance superior to several baseline approaches.
  arXiv  Detail & Related papers  (2022-11-07T10:18:31Z)
• Grounding Aleatoric Uncertainty in Unsupervised Environment Design [32.00797965770773]
  In partially-observable settings, optimal policies may depend on the ground-truth distribution over aleatoric parameters of the environment. We propose a minimax regret UED method that optimize the ground-truth utility function, even when the underlying training data is biased due to CICS.
  arXiv  Detail & Related papers  (2022-07-11T22:45:29Z)
• Lifelong Incremental Reinforcement Learning with Online Bayesian Inference [11.076005074172516]
  A long-lived reinforcement learning agent is to incrementally adapt its behavior as its environment changes. We propose LifeLong Reinforcement Learning (LLIRL), a new incremental algorithm for efficient lifelong adaptation to dynamic environments.
  arXiv  Detail & Related papers  (2020-07-28T13:23:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.