Why the Agent Made that Decision: Contrastive Explanation Learning for Reinforcement Learning

• URL: http://arxiv.org/abs/2411.16120v2
• Date: Wed, 06 Aug 2025 06:26:27 GMT
• Title: Why the Agent Made that Decision: Contrastive Explanation Learning for Reinforcement Learning
• Authors: Rui Zuo, Simon Khan, Zifan Wang, Garrett Ethan Katz, Qinru Qiu,
• Abstract summary: Reinforcement learning (RL) has demonstrated remarkable success in solving complex decision-making problems.<n>Existing explainable AI (xAI) approaches often fail to provide meaningful explanations for RL agents.<n>We propose a novel framework of contrastive learning to explain RL selected actions, named $textbfVisionMask$.
• Score: 11.068220265247385
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) has demonstrated remarkable success in solving complex decision-making problems, yet its adoption in critical domains is hindered by the lack of interpretability in its decision-making processes. Existing explainable AI (xAI) approaches often fail to provide meaningful explanations for RL agents, particularly because they overlook the contrastive nature of human reasoning--answering "why this action instead of that one?". To address this gap, we propose a novel framework of contrastive learning to explain RL selected actions, named $\textbf{VisionMask}$. VisionMask is trained to generate explanations by explicitly contrasting the agent's chosen action with alternative actions in a given state using a self-supervised manner. We demonstrate the efficacy of our method through experiments across diverse RL environments, evaluating it in terms of faithfulness, robustness, and complexity. Our results show that VisionMask significantly improves human understanding of agent behavior while maintaining accuracy and fidelity. Furthermore, we present examples illustrating how VisionMask can be used for counterfactual analysis. This work bridges the gap between RL and xAI, paving the way for safer and more interpretable RL systems.

Related papers

• Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning [87.7836502955847]
  We propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning.<n>Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood.<n>We introduce CoVo, an intrinsic reward mechanism that integrates Consistency and Volatility via a robust vector-space aggregation strategy.
  arXiv  Detail & Related papers  (2025-06-10T12:40:39Z)
• Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning [52.83539473110143]
  We introduce a novel structure-oriented analysis method to help Large Language Models (LLMs) better understand a question. To further improve the reliability in complex question-answering tasks, we propose a multi-agent reasoning system, Structure-oriented Autonomous Reasoning Agents (SARA) Extensive experiments verify the effectiveness of the proposed reasoning system. Surprisingly, in some cases, the system even surpasses few-shot methods.
  arXiv  Detail & Related papers  (2024-10-18T05:30:33Z)
• ID-Guard: A Universal Framework for Combating Facial Manipulation via Breaking Identification [60.73617868629575]
  misuse of deep learning-based facial manipulation poses a significant threat to civil rights. To prevent this fraud at its source, proactive defense has been proposed to disrupt the manipulation process. This paper proposes a universal framework for combating facial manipulation, termed ID-Guard.
  arXiv  Detail & Related papers  (2024-09-20T09:30:08Z)
• Causal State Distillation for Explainable Reinforcement Learning [16.998047658978482]
  Reinforcement learning (RL) is a powerful technique for training intelligent agents, but understanding why these agents make specific decisions can be challenging. Various approaches have been explored to address this problem, with one promising avenue being reward decomposition (RD) RD is appealing as it sidesteps some of the concerns associated with other methods that attempt to rationalize an agent's behaviour in a post-hoc manner. We present an extension of RD that goes beyond sub-rewards to provide more informative explanations.
  arXiv  Detail & Related papers  (2023-12-30T00:01:22Z)
• MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learning [40.7452827298478]
  We introduce MaDi, a novel algorithm that learns to mask distractions by the reward signal only. In MaDi, the conventional actor-critic structure of deep reinforcement learning agents is complemented by a small third sibling, the Masker. Our algorithm improves the agent's focus with useful masks, while its efficient Masker network only adds 0.2% more parameters to the original structure.
  arXiv  Detail & Related papers  (2023-12-23T20:11:05Z)
• Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning [53.00683059396803]
  Mask image model (MIM) has been widely used due to its simplicity and effectiveness in recovering original information from masked images. We propose a decision-based MIM that utilizes reinforcement learning (RL) to automatically search for optimal image masking ratio and masking strategy. Our approach has a significant advantage over alternative self-supervised methods on the task of neuron segmentation.
  arXiv  Detail & Related papers  (2023-10-06T10:40:46Z)
• Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning [69.19840497497503]
  It is argued that the commonly used action matching principle is more like an explanation of deep neural networks (DNNs) than the interpretation of RL agents. We propose to consider rewards, the essential objective of RL agents, as the essential objective of interpreting RL agents. We verify and evaluate our method on the Atari 2600 games as well as Duckietown, a challenging self-driving car simulator environment.
  arXiv  Detail & Related papers  (2023-09-04T09:09:54Z)
• MA2CL:Masked Attentive Contrastive Learning for Multi-Agent Reinforcement Learning [128.19212716007794]
  We propose an effective framework called textbfMulti-textbfAgent textbfMasked textbfAttentive textbfContrastive textbfLearning (MA2CL) MA2CL encourages learning representation to be both temporal and agent-level predictive by reconstructing the masked agent observation in latent space. Our method significantly improves the performance and sample efficiency of different MARL algorithms and outperforms other methods in various vision-based and state-based scenarios.
  arXiv  Detail & Related papers  (2023-06-03T05:32:19Z)
• Hard Patches Mining for Masked Image Modeling [52.46714618641274]
  Masked image modeling (MIM) has attracted much research attention due to its promising potential for learning scalable visual representations. We propose Hard Patches Mining (HPM), a brand-new framework for MIM pre-training.
  arXiv  Detail & Related papers  (2023-04-12T15:38:23Z)
• GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations [0.7874708385247353]
  We propose a novel but simple method to generate counterfactual explanations for RL agents. Our method is fully model-agnostic and we demonstrate that it outperforms the only previous method in several computational metrics.
  arXiv  Detail & Related papers  (2023-02-24T15:29:43Z)
• Masked Autoencoding for Scalable and Generalizable Decision Making [93.84855114717062]
  MaskDP is a simple and scalable self-supervised pretraining method for reinforcement learning and behavioral cloning. We find that a MaskDP model gains the capability of zero-shot transfer to new BC tasks, such as single and multiple goal reaching.
  arXiv  Detail & Related papers  (2022-11-23T07:04:41Z)
• Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities [2.0341936392563063]
  Most explanation methods for AI are focused on developers and expert users. Counterfactual explanations offer users advice on what can be changed in the input for the output of the black-box model to change. Counterfactuals are user-friendly and provide actionable advice for achieving the desired output from the AI system.
  arXiv  Detail & Related papers  (2022-10-21T09:50:53Z)
• Exploring Target Representations for Masked Autoencoders [78.57196600585462]
  We show that a careful choice of the target representation is unnecessary for learning good representations. We propose a multi-stage masked distillation pipeline and use a randomly model as the teacher. A proposed method to perform masked knowledge distillation with bootstrapped teachers (dBOT) outperforms previous self-supervised methods by nontrivial margins.
  arXiv  Detail & Related papers  (2022-09-08T16:55:19Z)
• Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning [92.18524491615548]
  Contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL) We study how RL can be empowered by contrastive learning in a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions. Under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs.
  arXiv  Detail & Related papers  (2022-07-29T17:29:08Z)
• Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies [79.60322329952453]
  We show how to develop interpretable representations of how agents make decisions. By understanding the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
  arXiv  Detail & Related papers  (2022-03-14T17:40:42Z)
• Self-supervised Transformer for Deepfake Detection [112.81127845409002]
  Deepfake techniques in real-world scenarios require stronger generalization abilities of face forgery detectors. Inspired by transfer learning, neural networks pre-trained on other large-scale face-related tasks may provide useful features for deepfake detection. In this paper, we propose a self-supervised transformer based audio-visual contrastive learning method.
  arXiv  Detail & Related papers  (2022-03-02T17:44:40Z)
• Mask or Non-Mask? Robust Face Mask Detector via Triplet-Consistency Representation Learning [23.062034116854875]
  In the absence of vaccines or medicines to stop COVID-19, one of the effective methods to slow the spread of the coronavirus is to wear a face mask. To mandate the use of face masks or coverings in public areas, additional human resources are required, which is tedious and attention-intensive. We propose a face mask detection framework that uses the context attention module to enable the effective attention of the feed-forward convolution neural network.
  arXiv  Detail & Related papers  (2021-10-01T16:44:06Z)
• Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning [9.49864824780503]
  We propose Mask-Attention A3C (Mask A3C), which introduces an attention mechanism into Asynchronous Advantage Actor-Critic (A3C) A3C consists of a feature extractor that extracts features from an image, a policy branch that outputs the policy, and a value branch that outputs the state value. We visualized mask-attention maps for games on the Atari 2600 and found we could easily analyze the reasons behind an agent's decision-making.
  arXiv  Detail & Related papers  (2021-03-06T08:38:12Z)
• Explainability in Deep Reinforcement Learning [68.8204255655161]
  We review recent works in the direction to attain Explainable Reinforcement Learning (XRL) In critical situations where it is essential to justify and explain the agent's behaviour, better explainability and interpretability of RL models could help gain scientific insight on the inner workings of what is still considered a black box.
  arXiv  Detail & Related papers  (2020-08-15T10:11:42Z)
• Explainable Reinforcement Learning: A Survey [0.0]
  Explainable Artificial Intelligence (XAI) has gained increased traction over the last few years. XAI models exhibit one detrimential characteristic: a performance-transparency trade-off. This survey attempts to address this gap by offering an overview of Explainable Reinforcement Learning (XRL) methods.
  arXiv  Detail & Related papers  (2020-05-13T10:52:49Z)
• Self-Supervised Discovering of Interpretable Features for Reinforcement Learning [40.52278913726904]
  We propose a self-supervised interpretable framework for deep reinforcement learning. A self-supervised interpretable network (SSINet) is employed to produce fine-grained attention masks for highlighting task-relevant information. We verify and evaluate our method on several Atari 2600 games as well as Duckietown, which is a challenging self-driving car simulator environment.
  arXiv  Detail & Related papers  (2020-03-16T08:26:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.