Behaviour Discovery and Attribution for Explainable Reinforcement Learning

• URL: http://arxiv.org/abs/2503.14973v1
• Date: Wed, 19 Mar 2025 08:06:00 GMT
• Title: Behaviour Discovery and Attribution for Explainable Reinforcement Learning
• Authors: Rishav Rishav, Somjit Nath, Vincent Michalski, Samira Ebrahimi Kahou,
• Abstract summary: We propose a framework for behavior discovery and action attribution to behaviors in offline RL trajectories.<n>Our method identifies meaningful behavioral segments, enabling more precise and granular explanations.<n>This approach is adaptable across diverse environments with minimal modifications.
• Score: 6.123880364445758
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Explaining the decisions made by reinforcement learning (RL) agents is critical for building trust and ensuring reliability in real-world applications. Traditional approaches to explainability often rely on saliency analysis, which can be limited in providing actionable insights. Recently, there has been growing interest in attributing RL decisions to specific trajectories within a dataset. However, these methods often generalize explanations to long trajectories, potentially involving multiple distinct behaviors. Often, providing multiple more fine grained explanations would improve clarity. In this work, we propose a framework for behavior discovery and action attribution to behaviors in offline RL trajectories. Our method identifies meaningful behavioral segments, enabling more precise and granular explanations associated with high level agent behaviors. This approach is adaptable across diverse environments with minimal modifications, offering a scalable and versatile solution for behavior discovery and attribution for explainable RL.

Related papers

• Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains [92.36624674516553]
  Reinforcement learning with verifiable rewards (RLVR) has demonstrated significant success in enhancing mathematical reasoning and coding performance of large language models (LLMs) We investigate the effectiveness and scalability of RLVR across diverse real-world domains including medicine, chemistry, psychology, economics, and education. We utilize a generative scoring technique that yields soft, model-based reward signals to overcome limitations posed by binary verifications.
  arXiv  Detail & Related papers  (2025-03-31T08:22:49Z)
• Analyzing Adversarial Inputs in Deep Reinforcement Learning [53.3760591018817]
  We present a comprehensive analysis of the characterization of adversarial inputs, through the lens of formal verification. We introduce a novel metric, the Adversarial Rate, to classify models based on their susceptibility to such perturbations. Our analysis empirically demonstrates how adversarial inputs can affect the safety of a given DRL system with respect to such perturbations.
  arXiv  Detail & Related papers  (2024-02-07T21:58:40Z)
• Causal State Distillation for Explainable Reinforcement Learning [16.998047658978482]
  Reinforcement learning (RL) is a powerful technique for training intelligent agents, but understanding why these agents make specific decisions can be challenging. Various approaches have been explored to address this problem, with one promising avenue being reward decomposition (RD) RD is appealing as it sidesteps some of the concerns associated with other methods that attempt to rationalize an agent's behaviour in a post-hoc manner. We present an extension of RD that goes beyond sub-rewards to provide more informative explanations.
  arXiv  Detail & Related papers  (2023-12-30T00:01:22Z)
• A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories [122.11358440078581]
  offline imitation is useful in real-world scenarios where arbitrary interactions are costly and expert actions are unavailable. We propose Trajectory-Aware Learning from Observations (TAILO) to solve MDPs where only task-specific expert states and task-agnostic non-expert state-action pairs are available.
  arXiv  Detail & Related papers  (2023-11-02T15:41:09Z)
• Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning [69.19840497497503]
  It is argued that the commonly used action matching principle is more like an explanation of deep neural networks (DNNs) than the interpretation of RL agents. We propose to consider rewards, the essential objective of RL agents, as the essential objective of interpreting RL agents. We verify and evaluate our method on the Atari 2600 games as well as Duckietown, a challenging self-driving car simulator environment.
  arXiv  Detail & Related papers  (2023-09-04T09:09:54Z)
• Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare [38.42691031505782]
  We propose a form of linear Q-function decomposition induced by factored action spaces. Our approach can help an agent make more accurate inferences within underexplored regions of the state-action space.
  arXiv  Detail & Related papers  (2023-05-02T19:13:10Z)
• RACCER: Towards Reachable and Certain Counterfactual Explanations for Reinforcement Learning [2.0341936392563063]
  We propose RACCER, the first-specific approach to generating counterfactual explanations for the behavior of RL agents. We use a tree search to find the most suitable counterfactuals based on the defined properties. We evaluate RACCER in two tasks as well as conduct a user study to show that RL-specific counterfactuals help users better understand agents' behavior.
  arXiv  Detail & Related papers  (2023-03-08T09:47:00Z)
• Disentangled Representation Learning [46.51815065323667]
  Disentangled Representation Learning (DRL) aims to learn a model capable of identifying and disentangling the underlying factors hidden in the observable data in representation form. We comprehensively investigate DRL from various aspects including motivations, definitions, methodologies, evaluations, applications, and model designs.
  arXiv  Detail & Related papers  (2022-11-21T18:14:38Z)
• Explainable Reinforcement Learning via Model Transforms [18.385505289067023]
  We argue that even if the underlying Markov Decision Process is not fully known, it can nevertheless be exploited to automatically generate explanations. We suggest using formal MDP abstractions and transforms, previously used in the literature for expediting the search for optimal policies, to automatically produce explanations.
  arXiv  Detail & Related papers  (2022-09-24T13:18:06Z)
• Beyond Rewards: a Hierarchical Perspective on Offline Multiagent Behavioral Analysis [14.656957226255628]
  We introduce a model-agnostic method for discovery of behavior clusters in multiagent domains. Our framework makes no assumption about agents' underlying learning algorithms, does not require access to their latent states or models, and can be trained using entirely offline observational data.
  arXiv  Detail & Related papers  (2022-06-17T23:07:33Z)
• Explainability in Deep Reinforcement Learning [68.8204255655161]
  We review recent works in the direction to attain Explainable Reinforcement Learning (XRL) In critical situations where it is essential to justify and explain the agent's behaviour, better explainability and interpretability of RL models could help gain scientific insight on the inner workings of what is still considered a black box.
  arXiv  Detail & Related papers  (2020-08-15T10:11:42Z)
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning [90.93035276307239]
  We propose an information theoretic regularization objective and an annealing-based optimization method to achieve better generalization ability in RL agents. We demonstrate the extreme generalization benefits of our approach in different domains ranging from maze navigation to robotic tasks. This work provides a principled way to improve generalization in RL by gradually removing information that is redundant for task-solving.
  arXiv  Detail & Related papers  (2020-08-03T02:24:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.