Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness

• URL: http://arxiv.org/abs/2605.23146v1
• Date: Fri, 22 May 2026 01:51:41 GMT
• Title: Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness
• Abstract summary: We present a proof-of-concept implementation of an infra-Bayesian reinforcement learning architecture for finite-outcome stateless decision problems.<n>Our agent maintains a set of hypotheses, updates them using infra-Bayesian conditioning, and selects actions by maximizing worst-case expected value.<n>Our results provide a step towards reinforcement learning agents that remain robust under model misspecification and policy-dependent uncertainty.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Classical reinforcement learning assumes the agent interacts with a fixed environment whose behavior does not depend on the agent's policy. This assumption breaks down in non-realizable settings where other actors might anticipate the agent's behavior, including environments crucial to AI safety, where the agent interacts with predictors, humans, other AI agents, and institutions. In such settings, the agent's model class fails to capture the world in which it operates. Under such misspecification, classical Bayesian methods can produce confidently wrong posteriors, unreliable decisions, and unbounded regret, as realizability fails to obtain. Infra-Bayesianism is a decision-theoretic framework that addresses these failures by distinguishing ordinary probabilistic uncertainty, where priors can be reasonably chosen, from Knightian uncertainty, where no grounds exist for the construction of such a prior. It does so by evaluating actions on their worst-case outcomes, rather than from posterior expectations or weighted averaging. We present the first proof-of-concept implementation of an infra-Bayesian reinforcement learning architecture for finite-outcome stateless decision problems. Our agent maintains a set of imprecise hypotheses, updates them using infra-Bayesian conditioning, and selects actions by maximizing worst-case expected value. We apply this implementation of the infra-Bayesian maximin decision process to an environment with Knightian uncertainty, and demonstrate a lower worst-case regret as compared to classical reinforcement learning agents. We also investigate Newcomb's problem and show that the infra-Bayesian agent picks the optimal strategy, outperforming classical decision theory agents. Our results provide a step towards reinforcement learning agents that remain robust under model misspecification and policy-dependent uncertainty.

Related papers

• Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning [63.24624171679711]
  Large language model (LLM)-based agents often make suboptimal tool-use decisions.<n>We propose TRUST, which incorporates uncertainty quantification into reward design as a repulsive force for maintaining uncertainty separation.
  arXiv  Detail & Related papers  (2026-06-05T07:08:34Z)
• Semantic Reward Collapse and the Preservation of Epistemic Integrity in Adaptive AI Systems [0.0]
  Recent advances in reinforcement learning have substantially improved the usability, coherence, and safety of large language models.<n> recurring behaviors such as performative certainty suggest unresolved structural issues within scalarized preference optimization systems.<n>We propose Semantic Reward Collapse (SRC), the compression of semantically distinct forms of evaluative dissatisfaction into generalized optimization signals.
  arXiv  Detail & Related papers  (2026-05-12T17:03:26Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty [1.6868147729303773]
  We prove "selection theorems" showing that low "average-case regret" forces an agent to implement a predictive, structured internal state.<n>We show that regret bounds limit probability mass on suboptimal bets, enforcing the predictive distinctions needed to separate high-margin outcomes.
  arXiv  Detail & Related papers  (2026-03-03T00:47:58Z)
• Agency and Architectural Limits: Why Optimization-Based Systems Cannot Be Norm-Responsive [0.0]
  AI systems are increasingly deployed in high-stakes contexts under the assumption they can be governed by norms.<n>This paper demonstrates that the assumption is formally invalid for optimization-based systems.
  arXiv  Detail & Related papers  (2026-02-26T17:16:17Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery [53.08822154199948]
  Unsupervised Environment Design (UED) methods have gained recent attention as their adaptive curricula promise to enable agents to be robust to in- and out-of-distribution tasks. This work investigates how existing UED methods select training environments, focusing on task prioritisation metrics. We develop a method that directly trains on scenarios with high learnability.
  arXiv  Detail & Related papers  (2024-08-27T14:31:54Z)
• Pure Exploration under Mediators' Feedback [63.56002444692792]
  Multi-armed bandits are a sequential-decision-making framework, where, at each interaction step, the learner selects an arm and observes a reward. We consider the scenario in which the learner has access to a set of mediators, each of which selects the arms on the agent's behalf according to a and possibly unknown policy. We propose a sequential decision-making strategy for discovering the best arm under the assumption that the mediators' policies are known to the learner.
  arXiv  Detail & Related papers  (2023-08-29T18:18:21Z)
• Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty [54.88405167739227]
  We present HAICU, a method for heterogeneous-agent trajectory forecasting that explicitly incorporates agents' class probabilities. We additionally present PUP, a new challenging real-world autonomous driving dataset. We demonstrate that incorporating class probabilities in trajectory forecasting significantly improves performance in the face of uncertainty.
  arXiv  Detail & Related papers  (2021-04-26T10:28:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.