Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning

• URL: http://arxiv.org/abs/2607.08647v1
• Date: Thu, 09 Jul 2026 16:18:16 GMT
• Title: Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning
• Abstract summary: Inverse reinforcement learning provides a principled way to infer objectives from human feedback.<n>We show how different feedback modalities constrain rewards, showing that, in the unlimited-data regime, comparisons impose strictly stronger global constraints than other modalities.<n>We introduce a hierarchical machine teaching algorithm for reward learning that operates across multiple MDPs.
• Score: 14.145365060331516
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As autonomous agents are increasingly deployed across diverse operational contexts, aligning their behavior with human intent demands reward functions that remain robust to such changes rather than overfitting to any single environment. Inverse reinforcement learning (IRL) provides a principled way to infer such objectives from human feedback. However, existing analyses of optimal teaching approaches for IRL focus on single-environment, demonstration-only settings, leaving underexplored how heterogeneous feedback modalities and environment dynamics jointly constrain reward functions that generalize across multiple environments. Because demonstrations in one MDP entangle reward information with that environments specific structure, the resulting rewards frequently fail to generalize when the agent is deployed in a new setting. We first analyze how different feedback modalities constrain rewards, showing that, in the unlimited-data regime, comparisons impose strictly stronger global constraints than other modalities. Beyond this theoretical analysis, we introduce a hierarchical machine teaching algorithm for reward learning that operates across multiple MDPs. The algorithm first greedily selects informative environments that expose complementary reward constraints, then strategically queries low-cost feedback within those environments. Empirically, our method achieves substantially lower regret and stronger generalization to held-out environments than uniform teaching baselines under identical feedback budgets, demonstrating the importance of multi-environment, multi-modal teaching for learning dynamics-robust reward functions.

Related papers

• EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning [57.312638079204255]
  Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments.<n>We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal.
  arXiv  Detail & Related papers  (2026-08-06T15:54:36Z)
• Can RL Improve Generalization of LLM Agents? An Empirical Study [68.19349692042341]
  Reinforcement fine-tuning (RFT) has shown promise for training LLM agents to perform multi-turn decision-making based on environment feedback.<n>In real-world deployment, agents may operate in unseen environments with different background knowledge.<n>We show that RFT generalizes well across task difficulty within an environment, but exhibits weaker transfer to unseen environments.
  arXiv  Detail & Related papers  (2026-03-12T14:54:59Z)
• Learning Pareto-Optimal Rewards from Noisy Preferences: A Framework for Multi-Objective Inverse Reinforcement Learning [0.0]
  We introduce a theoretical framework for preference-based Multi-Objective Inverse Reinforcement Learning (MO-IRL), where human preferences are modeled as latent vector-valued reward functions.<n>Our results bridge the gap between practical alignment techniques and theoretical guarantees, providing a principled foundation for learning aligned behaviors.
  arXiv  Detail & Related papers  (2025-05-17T06:09:13Z)
• On Generalization Across Environments In Multi-Objective Reinforcement Learning [6.686583184622338]
  We formalize the concept of generalization in Multi-Objective Reinforcement Learning (MORL) and how it can be evaluated.<n>We contribute a novel benchmark featuring diverse multi-objective domains with parameterized environment configurations.<n>Our baseline evaluations of state-of-the-art MORL algorithms on this benchmark reveals limited generalization capabilities, suggesting significant room for improvement.
  arXiv  Detail & Related papers  (2025-03-02T08:50:14Z)
• Subtask-Aware Visual Reward Learning from Segmented Demonstrations [97.80917991633248]
  This paper introduces REDS: REward learning from Demonstration with Demonstrations, a novel reward learning framework.<n>We train a dense reward function conditioned on video segments and their corresponding subtasks to ensure alignment with ground-truth reward signals.<n>Our experiments show that REDS significantly outperforms baseline methods on complex robotic manipulation tasks in Meta-World.
  arXiv  Detail & Related papers  (2025-02-28T01:25:37Z)
• EvIL: Evolution Strategies for Generalisable Imitation Learning [33.745657379141676]
  In imitation learning (IL) expert demonstrations and the environment we want to deploy our learned policy in aren't exactly the same. Compared to policy-centric approaches to IL like cloning, reward-centric approaches like inverse reinforcement learning (IRL) often better replicate expert behaviour in new environments. We find that modern deep IL algorithms frequently recover rewards which induce policies far weaker than the expert, even in the same environment the demonstrations were collected in. We propose a novel evolution-strategies based method EvIL to optimise for a reward-shaping term that speeds up re-training in the target environment.
  arXiv  Detail & Related papers  (2024-06-15T22:46:39Z)
• Invariant Causal Imitation Learning for Generalizable Policies [87.51882102248395]
  We propose Invariant Causal Learning (ICIL) to learn an imitation policy. ICIL learns a representation of causal features that is disentangled from the specific representations of noise variables. We show that ICIL is effective in learning imitation policies capable of generalizing to unseen environments.
  arXiv  Detail & Related papers  (2023-11-02T16:52:36Z)
• Inferring Versatile Behavior from Demonstrations by Matching Geometric Descriptors [72.62423312645953]
  Humans intuitively solve tasks in versatile ways, varying their behavior in terms of trajectory-based planning and for individual steps. Current Imitation Learning algorithms often only consider unimodal expert demonstrations and act in a state-action-based setting. Instead, we combine a mixture of movement primitives with a distribution matching objective to learn versatile behaviors that match the expert's behavior and versatility.
  arXiv  Detail & Related papers  (2022-10-17T16:42:59Z)
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning [16.910911657616005]
  Training deep reinforcement learning agents on environments with multiple levels / scenes from the same task, has become essential for many applications. We argue that the sample variance for a multi-scene environment is best minimized by treating each scene as a distinct MDP. We also demonstrate that the true joint value function for a multi-scene environment, follows a multi-modal distribution which is not captured by traditional CNN / LSTM based critic networks.
  arXiv  Detail & Related papers  (2021-02-14T23:30:13Z)
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning [12.76337275628074]
  In this work, we propose a variational dynamic model based on the conditional variational inference to model the multimodality andgenerativeity. We derive an upper bound of the negative log-likelihood of the environmental transition and use such an upper bound as the intrinsic reward for exploration. Our method outperforms several state-of-the-art environment model-based exploration approaches.
  arXiv  Detail & Related papers  (2020-10-17T09:54:51Z)
• Ecological Reinforcement Learning [76.9893572776141]
  We study the kinds of environment properties that can make learning under such conditions easier. understanding how properties of the environment impact the performance of reinforcement learning agents can help us to structure our tasks in ways that make learning tractable.
  arXiv  Detail & Related papers  (2020-06-22T17:55:03Z)
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning [22.889059874754242]
  Training deep reinforcement learning agents on environments with multiple levels / scenes / conditions from the same task, has become essential for many applications. We propose a dynamic value estimation (DVE) technique for these multiple-MDP environments, motivated by the clustering effect observed in the value function distribution across different scenes.
  arXiv  Detail & Related papers  (2020-05-25T17:56:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.