Towards a Belief-Based World Model for LLM Agents

• URL: http://arxiv.org/abs/2609.00455v1
• Date: Mon, 31 Aug 2026 22:48:38 GMT
• Title: Towards a Belief-Based World Model for LLM Agents
• Abstract summary: World models are a promising way to enhance policy performance.<n>During inference, agents currently use world models to simulate the consequences of candidate actions.<n>We argue that simulation alone is an incomplete interface for decision-making under partial observability.
• Score: 14.419757676852383
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are being used as policies for autonomous decision-making and planning in many domains. Despite their strong reasoning capabilities, LLMs struggle with long-horizon tasks, especially under partial observability. World models are a promising way to enhance policy performance, both during training and inference. During inference, agents currently use world models to simulate the consequences of candidate actions before committing to an action, which can improve decision-making. However, we argue that simulation alone is an incomplete interface for decision-making under partial observability: simulation doesn't adequately capture uncertainty about the current state, which agents may need for accurate decision-making. We address this limitation with Belief-Based World Models (BB-WMs), which model and maintain a belief that LLMs can query to access information on what is known and uncertain about the current state. Before developing methods to learn accurate BB-WMs, we first ask a more fundamental question: does exposing a world model's belief directly to an LLM policy improve decision-making? Our results show that giving LLM agents access to world model beliefs improves task performance under partial observability, while remaining complementary to existing simulation-based world models. Code is released at https://github.com/skumar-ml/belief-world-models.

Related papers

• A Close Look At World Model Recovery In Supervised Fine-Tuned LLM Planners [5.334857433652785]
  Supervised fine-tuning (SFT) improves end-to-end classical planning in large language models (LLMs)<n>Do these models also learn to represent and reason about the planning problems they are solving?
  arXiv  Detail & Related papers  (2026-06-02T14:09:16Z)
• Reinforcement World Model Learning for LLM-based Agents [60.65003139516272]
  Reinforcement World Model Learning (RWML) is a self-conditioned method that learns action-supervised world models for LLM-based agents.<n>Our method aligns simulated next states produced by the model with realized next states observed from the environment.<n>We evaluate our method on ALFWorld and $2$ Bench and observe significant gains over the base model, despite being entirely self-supervised.
  arXiv  Detail & Related papers  (2026-02-05T16:30:08Z)
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs [72.08224879435762]
  textttLearn-to-Ask is a simulator-free framework for learning and deploying proactive dialogue agents.<n>Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service.
  arXiv  Detail & Related papers  (2025-10-29T12:08:07Z)
• R-WoM: Retrieval-augmented World Model For Computer-use Agents [15.812606459788471]
  Large Language Models (LLMs) can serve as world models to enhance agent decision-making in digital environments.<n>We probe two core capabilities of world models--future state prediction and reward estimation--through three tasks.<n>We propose the Retrieval-augmented World Model (R-WoM), which grounds simulations by incorporating factual, up-to-date knowledge retrieved from external tutorials.
  arXiv  Detail & Related papers  (2025-10-13T19:52:04Z)
• SimuRA: A World-Model-Driven Simulative Reasoning Architecture for General Goal-Oriented Agents [15.91448165400836]
  SimuRA is a goal-oriented architecture for generalized agentic reasoning.<n>We release ReasonerAgent-Web, a web-browsing agent built on SimuRA, as an open-source research demo.
  arXiv  Detail & Related papers  (2025-07-31T17:57:20Z)
• WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents [55.64361927346957]
  We propose a training-free "world alignment" that learns an environment's symbolic knowledge complementary to large language models (LLMs)<n>We also propose an RL-free, model-based agent "WALL-E 2.0" through the model-predictive control framework.<n> WALL-E 2.0 significantly outperforms existing methods on open-world challenges in Mars (Minecraft like) and ALFWorld (embodied indoor environments)
  arXiv  Detail & Related papers  (2025-04-22T10:58:27Z)
• AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretability [84.52205243353761]
  Recent work proposes using world models to generate controlled virtual environments in which AI agents can be tested before deployment.<n>We investigate ways of simplifying world models that remain agnostic to the AI agent under evaluation.
  arXiv  Detail & Related papers  (2025-04-06T20:35:44Z)
• Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data [46.65903742010956]
  We present the first large-scale quantitative evaluation of state-of-the-art LLMs' ability to accurately simulate human behavior.<n>Using real-world data from 31,865 online shopping sessions containing 230,965 user actions, our evaluation reveals that prompt-based LLMs achieve only 11.86% accuracy in generating human actions.<n>We also showcase that strategies as simple as fine-tuning LLMs on real human click-through data augmented with synthesized reasoning traces can greatly enhance models' performance.
  arXiv  Detail & Related papers  (2025-03-26T17:33:27Z)
• Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models [54.38054999271322]
  We show that large language models (LLMs) don't update their beliefs as expected from the Bayesian framework.<n>We teach the LLMs to reason in a Bayesian manner by training them to mimic the predictions of the normative Bayesian model.<n>More generally, our results indicate that LLMs can effectively learn reasoning skills from examples and generalize those skills to new domains.
  arXiv  Detail & Related papers  (2025-03-21T20:13:04Z)
• Scaling Autonomous Agents via Automatic Reward Modeling And Planning [52.39395405893965]
  Large language models (LLMs) have demonstrated remarkable capabilities across a range of tasks.<n>However, they still struggle with problems requiring multi-step decision-making and environmental feedback.<n>We propose a framework that can automatically learn a reward model from the environment without human annotations.
  arXiv  Detail & Related papers  (2025-02-17T18:49:25Z)
• Evaluating World Models with LLM for Decision Making [14.029514959117067]
  World model emerges as a key module in decision making, where MuZero and Dreamer achieve remarkable successes in complex tasks. Recent work leverages Large Language Models (LLMs) as general world simulators to simulate the dynamics of the world due to their generalizability. This work proposes a comprehensive evaluation of the world models with LLMs from the decision making perspective.
  arXiv  Detail & Related papers  (2024-11-13T17:19:32Z)
• Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation [25.26545170310844]
  We present a World-model-augmented (WMA) web agent, which simulates the outcomes of its actions for better decision-making.<n>Experiments on WebArena and Mind2Web show that our world models improve agents' policy selection without training.
  arXiv  Detail & Related papers  (2024-10-17T05:37:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.