Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmark

• URL: http://arxiv.org/abs/2605.06066v1
• Date: Thu, 07 May 2026 11:53:56 GMT
• Title: Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmark
• Abstract summary: We introduce MTG-Causal-RL, a Gymnasium benchmark built on Magic: The Gathering.<n>Every episode exposes causal variables, SCM-predicted intervention effects, and per-factor credit traces.<n>We propose Causal Graph-Factored Advantage PPO (CGFA-PPO) as a reference causal agent.
• Score: 35.748383442073276
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Causal reinforcement learning (RL) lacks benchmarks for complex systems that combine sequential decision making, hidden information, large masked action spaces, and explicit causal structure. We introduce MTG-Causal-RL, a Gymnasium benchmark built on Magic: The Gathering with a 3,077-dimensional partial observation, a 478-action masked discrete action space, five competitive Standard archetypes, three reward schemes, and a hand-specified Structural Causal Model (SCM) over strategic variables. Every episode exposes causal variables, SCM-predicted intervention effects, and per-factor credit traces, making causal credit assignment, leave-one-out cross-archetype transfer, and policy auditability first-class metrics. We adapt a panel of reference baselines: random, heuristic, masked PPO, a causal-world-model PPO variant, and an architecture-matched scalar control. We propose Causal Graph-Factored Advantage PPO (CGFA-PPO) as a reference causal agent that uses SCM parents of win probability as factor-aligned critic targets with an intervention-calibration loss. All comparisons use paired seeds, paired-bootstrap confidence intervals, and Holm-Bonferroni correction within pre-registered families. Masked PPO and CGFA-PPO reach competitive in-distribution win rates and exceed the random baseline; per-factor calibration trajectories and leave-one-out transfer gaps expose diagnostic structure that scalar win rate alone cannot. We release the benchmark, reference-baseline results, and full evaluation protocol openly. By coupling a strategically rich, partially observed domain with an explicit causal interface and statistical protocol, MTG-Causal-RL gives causal-RL, world-model, and LLM-agent research a shared testbed for questions current benchmarks cannot pose together: causal credit assignment under masked action spaces, structural transfer across archetypes, and SCM-grounded policy auditability.

Related papers

• Gated Decoupled Compositional Bandits: A Unified Theory of Contextual Bandits with Supervised-Calibrated Action Scaling and Pre-Execution Gating [0.0]
  Gated Decoupled Compositional Bandits (GDCB)<n>GDCB is a family of contextual bandit algorithms with three structural innovations.<n>We formalise this class of algorithms, prove four structural theorems characterising its statistical behaviour.
  arXiv  Detail & Related papers  (2026-08-22T14:58:25Z)
• SHARC: SHAP-Based Interpretability in Machine Learning Risk Models for Regulatory Capital under ICAAP and CCAR [0.0]
  This paper describes an explainability framework for the Hybrid GPR-HS architecture and its stress-testing extension.<n>It decomposes Stressed Value-at-Risk into baseline, mean-driven, and volatility-driven components, enabling transparent linkage between scenario design and capital outcomes.<n>Results establish SHARC as a regulator-aligned explainability layer that makes the Hybrid GPR-HS framework fully auditable and consistent with FRTB, ICAAP Pillar 2, and CCAR transparency requirements.
  arXiv  Detail & Related papers  (2026-07-06T16:45:11Z)
• How Useful is Causal Invariance for Domain Adaptation in Finite-Sample Settings? [58.740078141879984]
  Machine learning models often degrade when they are deployed on a target distribution that differs from the source distributions they were trained on.<n>Recent work in causality-based domain generalization has shown how shared causal structure can induce invariant predictors.<n>In this paper, we study when full or partial causal knowledge can provably improve supervised domain adaptation.
  arXiv  Detail & Related papers  (2026-06-10T21:07:49Z)
• Certified Policy Optimisation for Nested Causal Bandits via PAC-Bayes Risk [0.0]
  We formalise the problem class as Nested Contextual Causal Bandits (NCCBs)<n>Our main theoretical result is a causal PAC-Bayesian excess-risk bound that certifies any candidate deployment policy.
  arXiv  Detail & Related papers  (2026-05-28T11:39:51Z)
• DISA: Offline Importance Sampling for Distribution-Matching LLM-RL [56.9445657766829]
  We introduce DISA, which moves this calibration problem outside the RL loop.<n> DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate.<n>On two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline Flow.
  arXiv  Detail & Related papers  (2026-05-17T07:14:44Z)
• PRISM: A Geometric Risk Bound that Decomposes Drift into Scale, Shape, and Head [14.880821907124451]
  We propose PRISM, which exploits the linear output head of LLMs and the empirically near-isometric structure of their backbones.<n>The bound is calibrated for variant ranking and decomposes drift into three independently measurable axes.<n> PRISM ranks variants with mean Spearman correlations of 0.820 for post-training quantization and 0.831 for LoRA forgetting.
  arXiv  Detail & Related papers  (2026-05-12T06:40:34Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs [5.834576254792341]
  Chain-of-Thought (CoT) prompting can increase privacy risk by resurfacing personally identifiable information (PII) from the prompt into reasoning traces and outputs.<n>We study such direct, inference-time PII leakage using a model-agnostic framework.
  arXiv  Detail & Related papers  (2026-03-05T19:20:44Z)
• Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning [59.76691952347156]
  Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs)<n>Most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.<n>We propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL.
  arXiv  Detail & Related papers  (2026-01-26T21:38:20Z)
• Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning [60.00161035836637]
  Group Relative Policy Optimization has emerged as a promising critic-free reinforcement learning paradigm for reasoning tasks.<n>We introduce Outcome-grounded Advantage Reshaping (OAR), a fine-grained credit assignment mechanism that redistributes advantages based on how much each token influences the model's final answer.<n>OAR-G achieves comparable gains with negligible computational overhead, both significantly outperforming a strong GRPO baseline.
  arXiv  Detail & Related papers  (2026-01-12T10:48:02Z)
• ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking [84.07076200941474]
  ArenaRL is a reinforcement learning paradigm that shifts from pointwise scalar scoring to intra-group relative ranking.<n>We construct an intra-group adversarial arena and devise a tournament-based ranking scheme to obtain stable advantage signals.<n>Experiments show that ArenaRL substantially outperforms standard RL baselines.
  arXiv  Detail & Related papers  (2026-01-10T08:43:07Z)
• Partial Action Replacement: Tackling Distribution Shift in Offline MARL [11.861550409939818]
  offline multi-agent reinforcement learning (MARL) is severely hampered by the challenge of evaluating out-of-distribution joint actions.<n>We develop Soft-Partial Conservative Q-Learning (SPaCQL) to mitigate OOD issue and dynamically weighting different PAR strategies.<n>Our theoretical results also indicate that SPaCQL adaptively addresses distribution shift using uncertainty-informed weights.
  arXiv  Detail & Related papers  (2025-11-10T20:56:58Z)
• Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents [90.45197506653341]
  Large language model (LLM) agents increasingly rely on external tools such as search engines to solve complex, multi-step problems.<n>The trajectories of search agents are structurally heterogeneous, where variations in the number, placement, and outcomes of search calls lead to fundamentally different answer directions and reward distributions.<n>Standard policy gradient methods, which use a single global baseline, suffer from what we identify and formalize as cross-stratum bias.<n>We propose Stratified GRPO, whose central component, Stratified Advantage Normalization (SAN), partitions trajectories into homogeneous strata based on their structural properties and computes advantages locally within each
  arXiv  Detail & Related papers  (2025-10-07T17:59:13Z)
• Structure-Augmented Reasoning Generation [23.587337743113228]
  Retrieval-Augmented Generation (RAG) systems fail at complex multi-hop reasoning because they rely on large language models to implicitly connect information from unstructured document collections.<n>This fundamental limitation stems from treating retrieved passages as independent context rather than recognizing the intricate relationships that enable coherent reasoning chains.<n>We introduce SARG, a post-retrieval framework that transforms traditional RAG pipelines by materializing explicit reasoning structures.
  arXiv  Detail & Related papers  (2025-06-10T02:22:32Z)
• Answering Causal Queries at Layer 3 with DiscoSCMs-Embracing Heterogeneity [0.0]
  This paper advocates for the Distribution-consistency Structural Causal Models (DiscoSCM) framework as a pioneering approach to counterfactual inference.
  arXiv  Detail & Related papers  (2023-09-17T17:01:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.