CAST: Game Solvers as Turn-Level Teachers for LLM Agents

• URL: http://arxiv.org/abs/2607.25308v1
• Date: Tue, 28 Jul 2026 05:39:01 GMT
• Title: CAST: Game Solvers as Turn-Level Teachers for LLM Agents
• Abstract summary: Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making.<n>Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate.<n>We propose CAST, which converts changes in a game solver's state value into solver advantages and injects them into RLVR as turn-level signals.
• Score: 78.2379284312811
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.

Related papers

• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• Reinforcement Learning without Ground-Truth Solutions can Improve LLMs [47.0243847990524]
  Reinforcement learning with verifiable rewards (RLVR) for training LLMs typically rely on ground-truth answers to assign rewards.<n>We introduce a textbfRanking-textbfinduced textbfVERifiable framework (RiVER) that trains LLMs on score-based optimization tasks without ground-truth solutions.
  arXiv  Detail & Related papers  (2026-06-25T17:59:36Z)
• Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning [50.464623632604976]
  In this work, we study RL-based training of vision-language models (VLMs) for long-horizon decision-making in Super Mario Land.<n>We propose an adapted variant of PPO with a lightweight turn-level critic, which substantially improves training stability and sample efficiency.<n>We introduce Odysseus, an open training framework for VLM agents, achieving substantial gains across multiple levels of the game.
  arXiv  Detail & Related papers  (2026-05-01T02:05:56Z)
• GAMEBoT: Transparent Assessment of LLM Reasoning in Games [54.49589494014147]
  GAMEBoT is a gaming arena designed for rigorous assessment of Large Language Models.<n>We benchmark 17 prominent LLMs across eight games, encompassing various strategic abilities and game characteristics.<n>Our results suggest that GAMEBoT presents a significant challenge, even when LLMs are provided with detailed CoT prompts.
  arXiv  Detail & Related papers  (2024-12-18T08:32:53Z)
• Dense Reward for Free in Reinforcement Learning from Human Feedback [64.92448888346125]
  We leverage the fact that the reward model contains more information than just its scalar output. We use these attention weights to redistribute the reward along the whole completion. Empirically, we show that it stabilises training, accelerates the rate of learning, and, in practical cases, may lead to better local optima.
  arXiv  Detail & Related papers  (2024-02-01T17:10:35Z)
• Scaling Laws for Imitation Learning in Single-Agent Games [28.257046559127875]
  We investigate whether carefully scaling up model and data size can bring similar improvements in the imitation learning setting for single-agent games.<n>We first demonstrate our findings on a variety of Atari games, and thereafter focus on the extremely challenging game of NetHack.<n>We find that IL loss and mean return scale smoothly with the compute budget and are strongly correlated, resulting in power laws for training compute-optimal IL agents.
  arXiv  Detail & Related papers  (2023-07-18T16:43:03Z)
• Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learning [105.7510838453122]
  We examine online safe multi-agent reinforcement learning using constrained Markov games. We develop an upper confidence reinforcement learning algorithm to solve this Lagrangian problem. Our algorithm updates the minimax decision primal variables via online mirror descent and the dual variable via projected gradient step.
  arXiv  Detail & Related papers  (2023-05-31T22:09:24Z)
• A Ranking Game for Imitation Learning [22.028680861819215]
  We treat imitation as a two-player ranking-based Stackelberg game between a $textitpolicy$ and a $textitreward$ function. This game encompasses a large subset of both inverse reinforcement learning (IRL) methods and methods which learn from offline preferences. We theoretically analyze the requirements of the loss function used for ranking policy performances to facilitate near-optimal imitation learning at equilibrium.
  arXiv  Detail & Related papers  (2022-02-07T19:38:22Z)
• Doubly Optimal No-Regret Online Learning in Strongly Monotone Games with Bandit Feedback [29.553652241608997]
  We study the class of textitsmooth and strongly monotone games and study optimal no-regret learning therein. We first construct a new bandit learning algorithm and show that it achieves the single-agent optimal regret of $tildeTheta(nsqrtT)$. Our results thus settle this open problem and contribute to the broad landscape of bandit game-theoretical learning.
  arXiv  Detail & Related papers  (2021-12-06T08:27:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.