Mutual Information Tracks Policy Coherence in Reinforcement Learning

• URL: http://arxiv.org/abs/2509.10423v1
• Date: Fri, 12 Sep 2025 17:24:20 GMT
• Title: Mutual Information Tracks Policy Coherence in Reinforcement Learning
• Authors: Cameron Reid, Wael Hafez, Amirhossein Nazeri,
• Abstract summary: Reinforcement Learning (RL) agents face degradation from sensor faults, actuator wear, and environmental shifts.<n>We present an information-theoretic framework that reveals both the fundamental dynamics of RL and provides practical methods for diagnosing deployment-time anomalies.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reinforcement Learning (RL) agents deployed in real-world environments face degradation from sensor faults, actuator wear, and environmental shifts, yet lack intrinsic mechanisms to detect and diagnose these failures. We present an information-theoretic framework that reveals both the fundamental dynamics of RL and provides practical methods for diagnosing deployment-time anomalies. Through analysis of state-action mutual information patterns in a robotic control task, we first demonstrate that successful learning exhibits characteristic information signatures: mutual information between states and actions steadily increases from 0.84 to 2.83 bits (238% growth) despite growing state entropy, indicating that agents develop increasingly selective attention to task-relevant patterns. Intriguingly, states, actions and next states joint mutual information, MI(S,A;S'), follows an inverted U-curve, peaking during early learning before declining as the agent specializes suggesting a transition from broad exploration to efficient exploitation. More immediately actionable, we show that information metrics can differentially diagnose system failures: observation-space, i.e., states noise (sensor faults) produces broad collapses across all information channels with pronounced drops in state-action coupling, while action-space noise (actuator faults) selectively disrupts action-outcome predictability while preserving state-action relationships. This differential diagnostic capability demonstrated through controlled perturbation experiments enables precise fault localization without architectural modifications or performance degradation. By establishing information patterns as both signatures of learning and diagnostic for system health, we provide the foundation for adaptive RL systems capable of autonomous fault detection and policy adjustment based on information-theoretic principles.

Related papers

• DODO: Causal Structure Learning with Budgeted Interventions [1.0323063834827415]
  We introduce DODO, an algorithm defining how an Agent can autonomously learn the causal structure of its environment.<n>Results show better performance for DODO, compared to observational approaches, in all but the most limited resource conditions.
  arXiv  Detail & Related papers  (2025-10-09T13:32:33Z)
• Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Rails [103.05296856071931]
  We identify the Alignment Tipping Process (ATP), a critical post-deployment risk unique to self-evolving Large Language Model (LLM) agents.<n>ATP arises when continual interaction drives agents to abandon alignment constraints established during training in favor of reinforced, self-interested strategies.<n>Our experiments show that alignment benefits erode rapidly under self-evolution, with initially aligned models converging toward unaligned states.
  arXiv  Detail & Related papers  (2025-10-06T14:48:39Z)
• From Physics to Machine Learning and Back: Part II - Learning and Observational Bias in PHM [52.64097278841485]
  Review examines how incorporating learning and observational biases through physics-informed modeling and data strategies can guide models toward physically consistent and reliable predictions.<n>Fast adaptation methods including meta-learning and few-shot learning are reviewed alongside domain generalization techniques.
  arXiv  Detail & Related papers  (2025-09-25T14:15:43Z)
• Hierarchical Reinforcement Learning Framework for Adaptive Walking Control Using General Value Functions of Lower-Limb Sensor Signals [0.40498500266986387]
  We explore the use of Hierarchical Reinforcement Learning to develop adaptive control strategies for lower-limb exoskeletons.<n>Our approach breaks down the complex task of exoskeleton control adaptation into a higher-level framework for terrain strategy adaptation and a lower-level framework for providing predictive information.<n>We investigated two methods for incorporating actual and predicted sensor signals into a policy network with the intent to improve the decision-making capacity of the control system.
  arXiv  Detail & Related papers  (2025-07-22T19:47:04Z)
• Rethinking Spatio-Temporal Anomaly Detection: A Vision for Causality-Driven Cybersecurity [22.491097360752903]
  We advocate for a causal learning perspective to advance anomaly detection in spatially distributed infrastructures.<n>We identify and formalize three key directions: causal graph profiling, multi-view fusion, and continual causal graph learning.<n>Our objective is to lay a new research trajectory toward scalable, adaptive, explainable, and spatially grounded anomaly detection systems.
  arXiv  Detail & Related papers  (2025-07-10T21:19:28Z)
• Ensuring Medical AI Safety: Interpretability-Driven Detection and Mitigation of Spurious Model Behavior and Associated Data [14.991686165405959]
  We show the applicability of the framework using four medical datasets across two modalities.<n>We successfully identify and unlearn these biases in VGG16, ResNet50, and contemporary Vision Transformer models.
  arXiv  Detail & Related papers  (2025-01-23T16:39:09Z)
• Semi-supervised Anomaly Detection via Adaptive Reinforcement Learning-Enabled Method with Causal Inference for Sensor Signals [15.249261198557218]
  Semi-supervised anomaly detection for sensor signals is critical in ensuring system reliability in smart manufacturing. This paper innovatively constructs a counterfactual causal reinforcement learning model, termed Triple-Assisted Causal Reinforcement Learning Anomaly Detector (Tri-CRLAD) Experimental results across seven diverse sensor signal datasets demonstrate that Tri-CRLAD outperforms nine state-of-the-art baseline methods.
  arXiv  Detail & Related papers  (2024-05-11T06:10:05Z)
• KGroot: Enhancing Root Cause Analysis through Knowledge Graphs and Graph Convolutional Neural Networks [14.336830860792707]
  KGroot uses event knowledge and the correlation between events to perform root cause reasoning. Experiments demonstrate KGroot can locate the root cause with accuracy of 93.5% top 3 potential causes in second-level.
  arXiv  Detail & Related papers  (2024-02-11T10:30:38Z)
• Analyzing Adversarial Inputs in Deep Reinforcement Learning [53.3760591018817]
  We present a comprehensive analysis of the characterization of adversarial inputs, through the lens of formal verification. We introduce a novel metric, the Adversarial Rate, to classify models based on their susceptibility to such perturbations. Our analysis empirically demonstrates how adversarial inputs can affect the safety of a given DRL system with respect to such perturbations.
  arXiv  Detail & Related papers  (2024-02-07T21:58:40Z)
• Multi-modal Causal Structure Learning and Root Cause Analysis [67.67578590390907]
  We propose Mulan, a unified multi-modal causal structure learning method for root cause localization. We leverage a log-tailored language model to facilitate log representation learning, converting log sequences into time-series data. We also introduce a novel key performance indicator-aware attention mechanism for assessing modality reliability and co-learning a final causal graph.
  arXiv  Detail & Related papers  (2024-02-04T05:50:38Z)
• Interpretable Imitation Learning with Dynamic Causal Relations [65.18456572421702]
  We propose to expose captured knowledge in the form of a directed acyclic causal graph. We also design this causal discovery process to be state-dependent, enabling it to model the dynamics in latent causal graphs. The proposed framework is composed of three parts: a dynamic causal discovery module, a causality encoding module, and a prediction module, and is trained in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-09-30T20:59:42Z)
• Causal Disentanglement Hidden Markov Model for Fault Diagnosis [55.90917958154425]
  We propose a Causal Disentanglement Hidden Markov model (CDHM) to learn the causality in the bearing fault mechanism. Specifically, we make full use of the time-series data and progressively disentangle the vibration signal into fault-relevant and fault-irrelevant factors. To expand the scope of the application, we adopt unsupervised domain adaptation to transfer the learned disentangled representations to other working environments.
  arXiv  Detail & Related papers  (2023-08-06T05:58:45Z)
• Causality-Based Multivariate Time Series Anomaly Detection [63.799474860969156]
  We formulate the anomaly detection problem from a causal perspective and view anomalies as instances that do not follow the regular causal mechanism to generate the multivariate data. We then propose a causality-based anomaly detection approach, which first learns the causal structure from data and then infers whether an instance is an anomaly relative to the local causal mechanism. We evaluate our approach with both simulated and public datasets as well as a case study on real-world AIOps applications.
  arXiv  Detail & Related papers  (2022-06-30T06:00:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.