Diagnosing Non-Markovian Observations in Reinforcement Learning via Prediction-Based Violation Scoring

• URL: http://arxiv.org/abs/2603.27389v1
• Date: Sat, 28 Mar 2026 19:42:27 GMT
• Title: Diagnosing Non-Markovian Observations in Reinforcement Learning via Prediction-Based Violation Scoring
• Abstract summary: Reinforcement learning algorithms assume that observations satisfy the Markov property.<n>Real-world sensors frequently violate this assumption through correlated noise, latency, or partial observability.<n>This paper introduces a prediction-based scoring method that quantifies non-Markovian structure in observation trajectories.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning algorithms assume that observations satisfy the Markov property, yet real-world sensors frequently violate this assumption through correlated noise, latency, or partial observability. Standard performance metrics conflate Markov breakdowns with other sources of suboptimality, leaving practitioners without diagnostic tools for such violations. This paper introduces a prediction-based scoring method that quantifies non-Markovian structure in observation trajectories. A random forest first removes nonlinear Markov-compliant dynamics; ridge regression then tests whether historical observations reduce prediction error on the residuals beyond what the current observation provides. The resulting score is bounded in [0, 1] and requires no causal graph construction. Evaluation spans six environments (CartPole, Pendulum, Acrobot, HalfCheetah, Hopper, Walker2d), three algorithms (PPO, A2C, SAC), controlled AR(1) noise at six intensity levels, and 10 seeds per condition. In post-hoc detection, 7 of 16 environment-algorithm pairs, primarily high-dimensional locomotion tasks, show significant positive monotonicity between noise intensity and the violation score (Spearman rho up to 0.78, confirmed under repeated-measures analysis); under training-time noise, 13 of 16 pairs exhibit statistically significant reward degradation. An inversion phenomenon is documented in low-dimensional environments where the random forest absorbs the noise signal, causing the score to decrease as true violations grow, a failure mode analyzed in detail. A practical utility experiment demonstrates that the proposed score correctly identifies partial observability and guides architecture selection, fully recovering performance lost to non-Markovian observations. Source code to reproduce all results is provided at https://github.com/NAVEENMN/Markovianes.

Related papers

• Evidence of Absence: Cross-Modal Abductive Risk Perception to Sustain World Models When Vision Fails [11.206758778146282]
  A structured world-state is designed to preserve prediction-critical content when perception degrades, but it presumes observations to populate it.<n>We address how to sustain the world model from a complementary modality by treating the absence of expected co-evidence as evidence of a hidden cause.<n>A microphone-array front-end estimates the bearing of engine and tire sources and extracts approach-rate evidence.
  arXiv  Detail & Related papers  (2026-08-15T00:38:17Z)
• Conformal calibration and look-elsewhere effect in anomaly detection for new-physics searches [0.0]
  Machine-learned anomaly detection is reshaping searches for new physics, but it has outrun the statistics used to interpret it.<n>We propose a calibration layer, built on conformal prediction, that turns any anomaly score into a defensible significance.
  arXiv  Detail & Related papers  (2026-06-11T18:00:02Z)
• Kurtosis-Guided Denoising Score Matching for Tabular Anomaly Detection [6.893608092072852]
  Denoising score matching (DSM) provides a way to learn data distributions by training a neural network to recover the score function.<n>We introduce kurtosis-based noise scaling (K-DSM), a per-feature scheme that sets noise levels from the shape of each marginal distribution.<n>On standard anomaly detection benchmarks, K-DSM achieves state-of-the-art performance in the semi-supervised setting.
  arXiv  Detail & Related papers  (2026-05-07T21:19:12Z)
• Variational Rectification Inference for Learning with Noisy Labels [74.85528327499662]
  We propose variational rectification inference (VRI) to formulate the adaptive rectification for loss functions.<n>VRI is constructed as a hierarchical Bayes by treating the rectifying vector as a latent variable.<n>By introducing the variational term in VRI, the conditional posterior is estimated accurately and avoids collapsing to a Dirac delta function.
  arXiv  Detail & Related papers  (2026-03-18T01:25:08Z)
• Noise-Robust Tiny Object Localization with Flows [63.60972031108944]
  We propose a noise-robust localization framework leveraging normalizing flows for flexible error modeling and uncertainty-guided optimization.<n>Our method captures complex, non-Gaussian prediction distributions through flow-based error modeling, enabling robust learning under noisy supervision.<n>An uncertainty-aware gradient modulation mechanism further suppresses learning from high-uncertainty, noise-prone samples, mitigating overfitting while stabilizing training.
  arXiv  Detail & Related papers  (2026-01-02T09:16:55Z)
• COGNOS: Universal Enhancement for Time Series Anomaly Detection via Constrained Gaussian-Noise Optimization and Smoothing [6.738487965771888]
  Constrained Gaussian-Noise Optimization and Smoothing (COGNOS) is a universal, model-agnostic enhancement framework.<n>We show that COGNOS is highly effective, delivering an average F-score uplift of 57.9% when applied to 12 diverse backbone models.<n>Our work reveals that directly regularizing output statistics is a powerful and generalizable strategy for significantly improving anomaly detection systems.
  arXiv  Detail & Related papers  (2025-11-10T09:43:48Z)
• Anomalous Decision Discovery using Inverse Reinforcement Learning [3.3675535571071746]
  Anomaly detection plays a critical role in Autonomous Vehicles (AVs) by identifying unusual behaviors through perception systems.<n>Current approaches, which often rely on predefined thresholds or supervised learning paradigms, exhibit reduced efficacy when confronted with unseen scenarios.<n>We present Trajectory-Reward Guided Adaptive Pre-training (TRAP), a novel IRL framework for anomaly detection.
  arXiv  Detail & Related papers  (2025-07-06T17:01:02Z)
• Quantifying First-Order Markov Violations in Noisy Reinforcement Learning: A Causal Discovery Approach [0.0]
  Reinforcement learning (RL) methods assume that each new observation completely reflects the environment's state.<n>In practice, partial observability or sensor/actuator noise often invalidates this assumption.<n>This paper proposes a systematic methodology for detecting such violations with a novel Markov Violation score (MVS)<n>MVS measures multi-step dependencies that emerge when noise or incomplete state information disrupts the Markov property.
  arXiv  Detail & Related papers  (2025-02-28T21:42:10Z)
• SoftPatch: Unsupervised Anomaly Detection with Noisy Data [67.38948127630644]
  This paper considers label-level noise in image sensory anomaly detection for the first time. We propose a memory-based unsupervised AD method, SoftPatch, which efficiently denoises the data at the patch level. Compared with existing methods, SoftPatch maintains a strong modeling ability of normal data and alleviates the overconfidence problem in coreset.
  arXiv  Detail & Related papers  (2024-03-21T08:49:34Z)
• Label Noise: Correcting the Forward-Correction [0.0]
  Training neural network classifiers on datasets with label noise poses a risk of overfitting them to the noisy labels. We propose an approach to tackling overfitting caused by label noise. Motivated by this observation, we propose imposing a lower bound on the training loss to mitigate overfitting.
  arXiv  Detail & Related papers  (2023-07-24T19:41:19Z)
• Doubly Stochastic Models: Learning with Unbiased Label Noises and Inference Stability [85.1044381834036]
  We investigate the implicit regularization effects of label noises under mini-batch sampling settings of gradient descent. We find such implicit regularizer would favor some convergence points that could stabilize model outputs against perturbation of parameters. Our work doesn't assume SGD as an Ornstein-Uhlenbeck like process and achieve a more general result with convergence of approximation proved.
  arXiv  Detail & Related papers  (2023-04-01T14:09:07Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)
• Neighborhood Collective Estimation for Noisy Label Identification and Correction [92.20697827784426]
  Learning with noisy labels (LNL) aims at designing strategies to improve model performance and generalization by mitigating the effects of model overfitting to noisy labels. Recent advances employ the predicted label distributions of individual samples to perform noise verification and noisy label correction, easily giving rise to confirmation bias. We propose Neighborhood Collective Estimation, in which the predictive reliability of a candidate sample is re-estimated by contrasting it against its feature-space nearest neighbors.
  arXiv  Detail & Related papers  (2022-08-05T14:47:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.