Why We Need World Models for AGI: Where LLMs Fail and How World Models May Outperform

• URL: http://arxiv.org/abs/2605.23972v1
• Date: Wed, 13 May 2026 12:55:34 GMT
• Title: Why We Need World Models for AGI: Where LLMs Fail and How World Models May Outperform
• Abstract summary: Large language models achieve strong performance in language generation and knowledge-intensive tasks.<n>We argue that these limitations may arise from an objective-level mismatch between sequence prediction and reasoning over latent environment dynamics.<n>We introduce Latent Dynamics Inference (LDI), a conceptual perspective that interprets language and multimodal observations as partial evidence of underlying transition dynamics.
• Score: 1.6746114653388389
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models achieve strong performance in language generation and knowledge-intensive tasks, yet remain limited in settings requiring causal reasoning, persistent state tracking, and long-horizon planning. We argue that these limitations may arise from an objective-level mismatch between sequence prediction and reasoning over latent environment dynamics. To formalize this distinction, we introduce Latent Dynamics Inference (LDI), a conceptual perspective that interprets language and multimodal observations as partial evidence of underlying transition dynamics. To empirically investigate this perspective, we introduce Flux, a sequential reasoning environment specified entirely through natural-language rules. As a proof-of-concept case study, the rules are first compiled into an explicit state-transition simulator, illustrating that structured latent transition dynamics can, in some cases, be operationally extracted from textual rule descriptions. This enables a controlled comparison between the LLMs operating purely over textual observations and reinforcement-learning agents trained directly within the extracted latent state space. Within this case study, agents operating with explicit access to the latent state space exhibit substantially more stable behavior in long-horizon gameplay, achieving an aggregate win rate of approximately 79% versus 11% for LLMs. Qualitative analysis further reveals failure modes consistent with unstable persistent state tracking, including invalid actions, state-tracking errors, and short-horizon reasoning failures. The complete implementation of the Flux environment available at https://github.com/FeisalAlaswad/FLUX-RL-Agent Within the evaluated setting, these results suggest that strong sequence prediction alone may struggle to support robust long-horizon dynamic reasoning without mechanisms for persistent state tracking and transition modeling

Related papers

• Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty [70.43119366710778]
  We introduce Agent-BRACE: Agent Belief state Representation via Abstraction and Confidence Estimation.<n>It is a method that decouples an LLM agent into a belief state model and a policy model, jointly optimized via reinforcement learning.<n>It achieves an average absolute improvement of +14.5% across long-horizon, partially observable language environments.
  arXiv  Detail & Related papers  (2026-05-12T02:37:04Z)
• Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models [154.39583176906893]
  Large language models (MLs) have advanced understanding, they remain highly prone to hallucinations in scenes.<n>We argue this stems from a failure intemporal monitoring, the ability to persistently track object dynamic identities, states, and relations over time.<n>Existing benchmarks obscure this deficit relying on single final-answer queries that can often be resolved via local visual cues or statistical priors.
  arXiv  Detail & Related papers  (2026-05-09T14:32:36Z)
• ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model [53.15040805435013]
  Vision-language models (VLMs) provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames.<n>We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance.
  arXiv  Detail & Related papers  (2026-03-23T17:59:42Z)
• KnowDiffuser: A Knowledge-Guided Diffusion Planner with LM Reasoning and Prior-Informed Trajectory Initialization [8.04458701181863]
  KnowDiffuser is a knowledge-guided motion planning framework.<n>It integrates semantic understanding of language models with the generative power of diffusion models.<n>It significantly outperforms existing planners in both open-loop and closed-loop evaluations.
  arXiv  Detail & Related papers  (2026-03-11T05:45:29Z)
• Let's Reward Step-by-Step: Step-Aware Contrastive Alignment for Vision-Language Navigation in Continuous Environments [38.97818584066075]
  Vision-Language Navigation in Continuous Environments (VLN-CE) requires agents to learn complex reasoning from long-horizon human interactions.<n>Current training paradigms struggle to balance generalization capability, error recovery and training stability.<n>We introduce Step-Aware Contrastive Alignment (SACA), a framework designed to extract dense supervision from imperfect trajectories.
  arXiv  Detail & Related papers  (2026-03-10T14:45:50Z)
• Neural Implicit Action Fields: From Discrete Waypoints to Continuous Functions for Vision-Language-Action Models [35.98187913967879]
  We propose a paradigm shift that reformulates action prediction from discrete waypoints to continuous action function regression.<n>Our approach achieves state-of-the-art results on CALVIN and LIBERO benchmarks across diverse backbones.
  arXiv  Detail & Related papers  (2026-03-02T11:48:24Z)
• Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction [2.5450067638785945]
  This study introduces Conditioned Comment Prediction (CCP), a task in which a model predicts how a user would comment on a given stimulus.<n>We evaluate open-weight 8B models (Llama3.1, Qwen3, Ministral) in English, German, and Luxembourgish language scenarios.
  arXiv  Detail & Related papers  (2026-02-26T08:40:21Z)
• From Observations to States: Latent Time Series Forecasting [65.98504021691666]
  We propose Latent Time Series Forecasting (LatentTSF), a novel paradigm that shifts TSF from observation regression to latent state prediction.<n>Specifically, LatentTSF employs an AutoEncoder to project observations at each time step into a higher-dimensional latent state space.<n>Our proposed latent objectives implicitly maximize mutual information between predicted latent states and ground-truth states and observations.
  arXiv  Detail & Related papers  (2026-01-30T20:39:44Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• CTRLS: Chain-of-Thought Reasoning via Latent State-Transition [57.51370433303236]
  Chain-of-thought (CoT) reasoning enables large language models to break down complex problems into interpretable intermediate steps.<n>We introduce groundingS, a framework that formulates CoT reasoning as a Markov decision process (MDP) with latent state transitions.<n>We show improvements in reasoning accuracy, diversity, and exploration efficiency across benchmark reasoning tasks.
  arXiv  Detail & Related papers  (2025-07-10T21:32:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.