Environment Parameter Gradient Theorem for Policy-Environment Co-Design in Reinforcement Learning

• URL: http://arxiv.org/abs/2607.12590v1
• Date: Tue, 14 Jul 2026 10:05:06 GMT
• Title: Environment Parameter Gradient Theorem for Policy-Environment Co-Design in Reinforcement Learning
• Abstract summary: Environment parameters can be tuned to shape the transition dynamics and costs experienced by the agent.<n>We establish an Environment Gradient Theorem -- a formal expression for the gradient of the value function with respect to environment parameters.<n>We develop a model-free algorithm that simultaneously learns the optimal policy and the environment parameters.
• Score: 0.4061135251278187
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) is traditionally concerned with learning a control policy for a fixed environment. In many engineering systems, however, the environment itself is alterable: physical or operational parameters can be tuned to shape the transition dynamics and costs experienced by the agent. This motivates jointly optimizing both the policy and the environment design parameters. To this end, we establish an Environment Parameter Gradient Theorem -- a formal expression for the gradient of the value function with respect to environment parameters. The key theoretical device is a generalized action-value function $Q_{π,ξ}(s,a,ζ)$, which comprises two copies of the environment parameters: $ζ$ governs the cost and transition dynamics at the current state--action pair, while $ξ$ governs the future rollouts. This decoupling yields a tractable closed-form gradient expression and is essential to the theorem's derivation. Building on this result, we develop a model-free algorithm that simultaneously learns the optimal policy and the environment parameters. We demonstrate the efficacy of our framework on a UAV network design problem, where the optimal UAV placement (environment parameters) and communication routes (governed by the policy) are learned jointly to minimize the total communication cost in the network.

Related papers

• EnvHarness: Awakening Static Worlds for Agent Learning [58.54600450547412]
  Environment Harness wraps a static environment to reshape its behavior without modifying the underlying logic.<n>EnvHarness applies across diverse domains while ensuring every reshaped environment retains its original verifier.<n>EnvRigger treats the target policy as a black box, observing its execution trajectories to synthesize EnvHarness components targeting diagnosed flaws.
  arXiv  Detail & Related papers  (2026-08-20T10:42:06Z)
• From Noise to Control: Parameterized Diffusion Policies [54.941343992110426]
  We propose a framework for learning diffusion policies conditioned on continuous parameters embedded in a learned behavior manifold.<n>Our approach enables smooth distances between known strategies and efficient adaptation to novel constraints without updating policy weights.
  arXiv  Detail & Related papers  (2026-05-29T20:21:50Z)
• PACE: Parameter Change for Unsupervised Environment Design [22.066871262223312]
  Unsupervised Environment Design offers promising paradigm for improving reinforcement learning generalization by adaptively shaping training environments.<n>Existing UED methods evaluate environments using indirect proxy signals such as regret, value-based errors, or Monte Carlo, which suffer from bias, high variance, or substantial computational overhead.<n>We propose Change Environment Design (PACE), which evaluates an environment through the policy change induced by training on that environment, directly grounding environment selection in realized learning progress.
  arXiv  Detail & Related papers  (2026-05-02T10:07:59Z)
• ProAct: Agentic Lookahead in Interactive Environments [56.50613398808361]
  ProAct is a framework that enables agents to internalize accurate lookahead reasoning through a two-stage training paradigm.<n>We introduce Grounded LookAhead Distillation (GLAD), where the agent undergoes supervised fine-tuning on trajectories derived from environment-based search.<n>We also propose the Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator designed to enhance policy-gradient algorithms.
  arXiv  Detail & Related papers  (2026-02-05T05:45:16Z)
• Capturing Visual Environment Structure Correlates with Control Performance [62.69063991419694]
  Existing proxy metrics focus on the representation's capacity to capture narrow aspects of the visual world.<n>We measure how well pretrained visual encoders support decoding of environment state.<n>We show that this probing accuracy strongly correlates with downstream policy performance.
  arXiv  Detail & Related papers  (2026-02-04T18:59:12Z)
• Learning Policy Representations for Steerable Behavior Synthesis [80.4542176039074]
  Given a Markov decision process (MDP), we seek to learn representations for a range of policies to facilitate behavior steering at test time.<n>We show that these representations can be approximated uniformly for a range of policies using a set-based architecture.<n>We use variational generative approach to induce a smooth latent space, and further shape it with contrastive learning so that latent distances align with differences in value functions.
  arXiv  Detail & Related papers  (2026-01-29T21:52:06Z)
• On the development of a practical Bayesian optimisation algorithm for expensive experiments and simulations with changing environmental conditions [0.0]
  This article extends Bayesian optimisation to the optimisation of systems in changing environments. The proposed algorithm is applied to a wind farm simulator with eight controllable and one environmental parameter.
  arXiv  Detail & Related papers  (2024-02-05T13:46:04Z)
• Reparameterized Policy Learning for Multimodal Trajectory Optimization [61.13228961771765]
  We investigate the challenge of parametrizing policies for reinforcement learning in high-dimensional continuous action spaces. We propose a principled framework that models the continuous RL policy as a generative model of optimal trajectories. We present a practical model-based RL method, which leverages the multimodal policy parameterization and learned world model.
  arXiv  Detail & Related papers  (2023-07-20T09:05:46Z)
• Constrained Environment Optimization for Prioritized Multi-Agent Navigation [11.473177123332281]
  This paper aims to consider the environment as a decision variable in a system-level optimization problem. We propose novel problems of unprioritized and prioritized environment optimization. We show, through formal proofs, under which conditions the environment can change while guaranteeing completeness.
  arXiv  Detail & Related papers  (2023-05-18T18:55:06Z)
• Environment Optimization for Multi-Agent Navigation [11.473177123332281]
  The goal of this paper is to consider the environment as a decision variable in a system-level optimization problem. We show, through formal proofs, under which conditions the environment can change while guaranteeing completeness. In order to accommodate a broad range of implementation scenarios, we include both online and offline optimization, and both discrete and continuous environment representations.
  arXiv  Detail & Related papers  (2022-09-22T19:22:16Z)
• Federated Reinforcement Learning with Environment Heterogeneity [30.797692838836277]
  We study a Federated Reinforcement Learning (FedRL) problem in which $n$ agents collaboratively learn a single policy without sharing the trajectories they collected during agent-environment interaction. We propose two federated RL algorithms, textttQAvg and textttPAvg.
  arXiv  Detail & Related papers  (2022-04-06T07:21:00Z)
• Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design [121.73425076217471]
  We propose Unsupervised Environment Design (UED), where developers provide environments with unknown parameters, and these parameters are used to automatically produce a distribution over valid, solvable environments. We call our technique Protagonist Antagonist Induced Regret Environment Design (PAIRED) Our experiments demonstrate that PAIRED produces a natural curriculum of increasingly complex environments, and PAIRED agents achieve higher zero-shot transfer performance when tested in highly novel environments.
  arXiv  Detail & Related papers  (2020-12-03T17:37:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.