Explainable Reinforcement Learning Agents Using World Models

• URL: http://arxiv.org/abs/2505.08073v1
• Date: Mon, 12 May 2025 21:18:31 GMT
• Title: Explainable Reinforcement Learning Agents Using World Models
• Authors: Madhuri Singh, Amal Alabdulkarim, Gennie Mansi, Mark O. Riedl,
• Abstract summary: We introduce a technique for using World Models to generate explanations for Model-Based Deep RL agents.<n>World Models predict how the world will change when actions are performed, allowing for the generation of counterfactual trajectories.<n>We show that explanations that show users what the world should have been like significantly increase their understanding of the agent policy.
• Score: 14.489344385644467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Explainable AI (XAI) systems have been proposed to help people understand how AI systems produce outputs and behaviors. Explainable Reinforcement Learning (XRL) has an added complexity due to the temporal nature of sequential decision-making. Further, non-AI experts do not necessarily have the ability to alter an agent or its policy. We introduce a technique for using World Models to generate explanations for Model-Based Deep RL agents. World Models predict how the world will change when actions are performed, allowing for the generation of counterfactual trajectories. However, identifying what a user wanted the agent to do is not enough to understand why the agent did something else. We augment Model-Based RL agents with a Reverse World Model, which predicts what the state of the world should have been for the agent to prefer a given counterfactual action. We show that explanations that show users what the world should have been like significantly increase their understanding of the agent policy. We hypothesize that our explanations can help users learn how to control the agents execution through by manipulating the environment.

Related papers

• Embodied AI Agents: Modeling the World [188.85697524284834]
  This paper describes our research on AI agents embodied in visual, virtual or physical forms.<n>We propose that the development of world models is central to reasoning and planning of embodied AI agents.<n>We also propose to learn the mental world model of users to enable better human-agent collaboration.
  arXiv  Detail & Related papers  (2025-06-27T16:05:34Z)
• Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents [64.75036903373712]
  Proposer-Agent-Evaluator is a learning system that enables foundation model agents to autonomously discover and practice skills in the wild.<n>At the heart of PAE is a context-aware task proposer that autonomously proposes tasks for the agent to practice with context information.<n>The success evaluation serves as the reward signal for the agent to refine its policies through RL.
  arXiv  Detail & Related papers  (2024-12-17T18:59:50Z)
• Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement [117.94654815220404]
  G"odel Agent is a self-evolving framework inspired by the G"odel machine.<n>G"odel Agent can achieve continuous self-improvement, surpassing manually crafted agents in performance, efficiency, and generalizability.
  arXiv  Detail & Related papers  (2024-10-06T10:49:40Z)
• Intention-aware policy graphs: answering what, how, and why in opaque agents [0.1398098625978622]
  Agents are a special kind of AI-based software in that they interact in complex environments and have increased potential for emergent behaviour. We propose a Probabilistic Graphical Model along with a pipeline for designing such model. We contribute measurements that evaluate the interpretability and reliability of explanations provided. This model can be constructed by taking partial observations of the agent's actions and world states.
  arXiv  Detail & Related papers  (2024-09-27T09:31:45Z)
• Semifactual Explanations for Reinforcement Learning [1.5320737596132754]
  Reinforcement Learning (RL) is a learning paradigm in which the agent learns from its environment through trial and error. Deep reinforcement learning (DRL) algorithms represent the agent's policies using neural networks, making their decisions difficult to interpret. Explaining the behaviour of DRL agents is necessary to advance user trust, increase engagement, and facilitate integration with real-life tasks.
  arXiv  Detail & Related papers  (2024-09-09T08:37:47Z)
• Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning [50.47568731994238]
  Key method for creating Artificial Intelligence (AI) agents is Reinforcement Learning (RL) This paper presents a general framework model for integrating and learning structured reasoning into AI agents' policies.
  arXiv  Detail & Related papers  (2023-12-22T17:57:57Z)
• The Rise and Potential of Large Language Model Based Agents: A Survey [91.71061158000953]
  Large language models (LLMs) are regarded as potential sparks for Artificial General Intelligence (AGI) We start by tracing the concept of agents from its philosophical origins to its development in AI, and explain why LLMs are suitable foundations for agents. We explore the extensive applications of LLM-based agents in three aspects: single-agent scenarios, multi-agent scenarios, and human-agent cooperation.
  arXiv  Detail & Related papers  (2023-09-14T17:12:03Z)
• GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations [0.7874708385247353]
  We propose a novel but simple method to generate counterfactual explanations for RL agents. Our method is fully model-agnostic and we demonstrate that it outperforms the only previous method in several computational metrics.
  arXiv  Detail & Related papers  (2023-02-24T15:29:43Z)
• Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning [85.86440477005523]
  We study more human-like RL agents which incorporate an established model of human-irrationality, the Rational Inattention (RI) model. RIRL models the cost of cognitive information processing using mutual information. We show that using RIRL yields a rich spectrum of new equilibrium behaviors that differ from those found under rational assumptions.
  arXiv  Detail & Related papers  (2022-01-18T20:54:00Z)
• Self-Initiated Open World Learning for Autonomous AI Agents [16.41396764793912]
  As more and more AI agents are used in practice, it is time to think about how to make these agents fully autonomous. This paper proposes a theoretic framework for this learning paradigm to promote the research of building Self-initiated Open world Learning agents.
  arXiv  Detail & Related papers  (2021-10-21T18:11:02Z)
• Explainability in Deep Reinforcement Learning [68.8204255655161]
  We review recent works in the direction to attain Explainable Reinforcement Learning (XRL) In critical situations where it is essential to justify and explain the agent's behaviour, better explainability and interpretability of RL models could help gain scientific insight on the inner workings of what is still considered a black box.
  arXiv  Detail & Related papers  (2020-08-15T10:11:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.