Certified Policy Optimisation for Nested Causal Bandits via PAC-Bayes Risk

• URL: http://arxiv.org/abs/2605.29788v1
• Date: Thu, 28 May 2026 11:39:51 GMT
• Title: Certified Policy Optimisation for Nested Causal Bandits via PAC-Bayes Risk
• Abstract summary: We formalise the problem class as Nested Contextual Causal Bandits (NCCBs)<n>Our main theoretical result is a causal PAC-Bayesian excess-risk bound that certifies any candidate deployment policy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Critical sequential decisions are rarely single-timescale: a strategic decision causally shapes the context in which every subsequent tactical choice is made; standard bandit and reinforcement-learning theory does not capture this causal coupling between timescales. We formalise the problem class as Nested Contextual Causal Bandits (NCCBs), a hierarchical SCM where each level's action sets the next level's context distribution, and propose Nested Causal Thompson Sampling (NCTS), which draws one mechanism-factorised belief per episode and acts recursively under it. Our main theoretical result is a causal PAC-Bayesian excess-risk bound that certifies any candidate deployment policy from historic data alone, off-policy and anytime, answering the deployment question: can we trust this agent here, and at what risk? Experiments on a hierarchical SCM show that, against a matched RFF-GP joint regression on the same function class, the factorised SCM-mechanism posterior transfers significantly better zero-shot under exogenous distribution shifts, the recursive meta-to-inner commit significantly dominates the joint-commit alternative in distribution, and the certificate significantly contracts as offline data accumulates. Combining these results, we establish progressive certified handover, a safe-deployment method: each timescale flips from a legacy controller to NCTS when gains can be certified, independently of the others.

Related papers

• MMD-Balls as Credal Sets: A PAC-Bayesian Framework for Epistemic Uncertainty in Test-Time Adaptation [0.0]
  Test-time adaptation methods improve model performance under distribution shift.<n>Test-time adaptation methods lack formal guarantees connecting shift magnitude to prediction reliability.<n>We develop a PAC-Bayesian framework yielding bounds explicitly parameterized by the maximum mean discrepancy (MMD) between source and target distributions.
  arXiv  Detail & Related papers  (2026-05-20T22:22:20Z)
• Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmark [35.748383442073276]
  We introduce MTG-Causal-RL, a Gymnasium benchmark built on Magic: The Gathering.<n>Every episode exposes causal variables, SCM-predicted intervention effects, and per-factor credit traces.<n>We propose Causal Graph-Factored Advantage PPO (CGFA-PPO) as a reference causal agent.
  arXiv  Detail & Related papers  (2026-05-07T11:53:56Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation [19.888565537472363]
  CoRM-RAG (Counterfactual Risk Minimization for RAG) is a framework that aligns retrieval with decision safety rather than mere similarity.<n>We introduce a Cognitive Perturbation Protocol to simulate user biases during training, which is then distilled into a lightweight Evidence Critic.<n>This scoring module learns to identify documents that possess sufficient evidential strength to steer the model toward correctness despite adversarial query perturbations.
  arXiv  Detail & Related papers  (2026-05-02T07:22:24Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding [59.16244104797919]
  We propose **Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC)**.<n>DARC frames response selection as distributionally robust, risk-sensitive decision making.<n>Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality.
  arXiv  Detail & Related papers  (2026-03-09T09:21:29Z)
• Concave Certificates: Geometric Framework for Distributionally Robust Risk and Complexity Analysis [0.7106986689736828]
  Distributionally Robust (DR) optimization aims to certify worst-case risk within a Wasserstein uncertainty set.<n>This paper introduces a novel geometric framework based on the least concave majorants of the growth rate function.<n>We extend this framework to complexity analysis, introducing a deterministic bound that complements standard statistical bound.
  arXiv  Detail & Related papers  (2026-01-04T00:24:43Z)
• Partial Action Replacement: Tackling Distribution Shift in Offline MARL [11.861550409939818]
  offline multi-agent reinforcement learning (MARL) is severely hampered by the challenge of evaluating out-of-distribution joint actions.<n>We develop Soft-Partial Conservative Q-Learning (SPaCQL) to mitigate OOD issue and dynamically weighting different PAR strategies.<n>Our theoretical results also indicate that SPaCQL adaptively addresses distribution shift using uncertainty-informed weights.
  arXiv  Detail & Related papers  (2025-11-10T20:56:58Z)
• Policy Evaluation in Distributional LQR [70.63903506291383]
  We provide a closed-form expression of the distribution of the random return. We show that this distribution can be approximated by a finite number of random variables. Using the approximate return distribution, we propose a zeroth-order policy gradient algorithm for risk-averse LQR.
  arXiv  Detail & Related papers  (2023-03-23T20:27:40Z)
• Policy Smoothing for Provably Robust Reinforcement Learning [109.90239627115336]
  We study the provable robustness of reinforcement learning against norm-bounded adversarial perturbations of the inputs. We generate certificates that guarantee that the total reward obtained by the smoothed policy will not fall below a certain threshold under a norm-bounded adversarial of perturbation the input.
  arXiv  Detail & Related papers  (2021-06-21T21:42:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.