Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Language

• URL: http://arxiv.org/abs/2511.14565v1
• Date: Tue, 18 Nov 2025 15:07:50 GMT
• Title: Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Language
• Authors: Minyoung Hwang, Alexandra Forsey-Smerek, Nathaniel Dennler, Andreea Bobu,
• Abstract summary: Masked Inverse Reinforcement Learning (Masked IRL) is a framework that uses large language models (LLMs) to combine the strengths of both input types.<n>In simulation and on a real robot, Masked IRL outperforms prior language-conditioned IRL methods by up to 15%.
• Score: 42.32510573712616
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robots can adapt to user preferences by learning reward functions from demonstrations, but with limited data, reward models often overfit to spurious correlations and fail to generalize. This happens because demonstrations show robots how to do a task but not what matters for that task, causing the model to focus on irrelevant state details. Natural language can more directly specify what the robot should focus on, and, in principle, disambiguate between many reward functions consistent with the demonstrations. However, existing language-conditioned reward learning methods typically treat instructions as simple conditioning signals, without fully exploiting their potential to resolve ambiguity. Moreover, real instructions are often ambiguous themselves, so naive conditioning is unreliable. Our key insight is that these two input types carry complementary information: demonstrations show how to act, while language specifies what is important. We propose Masked Inverse Reinforcement Learning (Masked IRL), a framework that uses large language models (LLMs) to combine the strengths of both input types. Masked IRL infers state-relevance masks from language instructions and enforces invariance to irrelevant state components. When instructions are ambiguous, it uses LLM reasoning to clarify them in the context of the demonstrations. In simulation and on a real robot, Masked IRL outperforms prior language-conditioned IRL methods by up to 15% while using up to 4.7 times less data, demonstrating improved sample-efficiency, generalization, and robustness to ambiguous language. Project page: https://MIT-CLEAR-Lab.github.io/Masked-IRL and Code: https://github.com/MIT-CLEAR-Lab/Masked-IRL

Related papers

• LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation [11.419077130835829]
  We introduce LACY (Language-Action Cycle), a unified framework that learns bidirectional mappings within a single vision-language model.<n>We show that LACY improves task success rates by 56.46% on average and yields more robust language-action grounding for robotic manipulation.
  arXiv  Detail & Related papers  (2025-11-04T04:02:51Z)
• Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics [29.557203948616678]
  Video-Language Critic is a reward model that can be trained on readily available cross-embodiment data.<n>Our model enables 2x more sample-efficient policy training on Meta-World tasks than a sparse reward only.
  arXiv  Detail & Related papers  (2024-05-30T12:18:06Z)
• Grounding Language Plans in Demonstrations Through Counterfactual Perturbations [25.19071357445557]
  Grounding the common-sense reasoning of Large Language Models (LLMs) in physical domains remains a pivotal yet unsolved problem for embodied AI. We show our approach improves the interpretability and reactivity of imitation learning through 2D navigation and simulated and real robot manipulation tasks.
  arXiv  Detail & Related papers  (2024-03-25T19:04:59Z)
• $L^*LM$: Learning Automata from Examples using Natural Language Oracles [24.830220966380352]
  $L*LM$ is an algorithm for learning DFAs from both demonstrations and natural language.<n>We observe a significant improvement in the data efficiency of learning DFAs from expert demonstrations.
  arXiv  Detail & Related papers  (2024-02-10T21:46:34Z)
• Preference-Conditioned Language-Guided Abstraction [24.626805570296064]
  We observe that how humans behave reveals how they see the world. In this work, we propose using language models (LMs) to query for those preferences directly given knowledge that a change in behavior has occurred. We demonstrate our framework's ability to construct effective preference-conditioned abstractions in simulated experiments, a user study, and on a real Spot robot performing mobile manipulation tasks.
  arXiv  Detail & Related papers  (2024-02-05T15:12:15Z)
• Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention [58.817405319722596]
  This work compares the self-attention of several large language models (LLMs) in different sizes to assess the effect of scaling and instruction tuning on language perception. Results show that scaling enhances the human resemblance and improves the effective attention by reducing the trivial pattern reliance, while instruction tuning does not. We also find that current LLMs are consistently closer to non-native than native speakers in attention, suggesting a sub-optimal language perception of all models.
  arXiv  Detail & Related papers  (2023-10-29T17:16:40Z)
• Self-Evolution Learning for Discriminative Language Model Pretraining [103.57103957631067]
  Self-Evolution learning (SE) is a simple and effective token masking and learning method. SE focuses on learning the informative yet under-explored tokens and adaptively regularizes the training by introducing a novel Token-specific Label Smoothing approach.
  arXiv  Detail & Related papers  (2023-05-24T16:00:54Z)
• Robustness of Demonstration-based Learning Under Limited Data Scenario [54.912936555876826]
  Demonstration-based learning has shown great potential in stimulating pretrained language models' ability under limited data scenario. Why such demonstrations are beneficial for the learning process remains unclear since there is no explicit alignment between the demonstrations and the predictions. In this paper, we design pathological demonstrations by gradually removing intuitively useful information from the standard ones to take a deep dive of the robustness of demonstration-based sequence labeling.
  arXiv  Detail & Related papers  (2022-10-19T16:15:04Z)
• Pre-Trained Language Models for Interactive Decision-Making [72.77825666035203]
  We describe a framework for imitation learning in which goals and observations are represented as a sequence of embeddings. We demonstrate that this framework enables effective generalization across different environments. For test tasks involving novel goals or novel scenes, initializing policies with language models improves task completion rates by 43.6%.
  arXiv  Detail & Related papers  (2022-02-03T18:55:52Z)
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents [111.33545170562337]
  We investigate the possibility of grounding high-level tasks, expressed in natural language, to a chosen set of actionable steps. We find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into low-level plans. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions.
  arXiv  Detail & Related papers  (2022-01-18T18:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.