How do World Models and Policies Compose in LLM Agents? A Joint Spectral and Behavioral Account

• URL: http://arxiv.org/abs/2608.30067v1
• Date: Sun, 30 Aug 2026 22:21:16 GMT
• Title: How do World Models and Policies Compose in LLM Agents? A Joint Spectral and Behavioral Account
• Abstract summary: We combine world-model training (next-state prediction) and policy training (reward)<n>We find effective world-model updates are low-rank and share an input-feature subspace with policy updates.<n>We ask: does policy training preserve world knowledge as well as it could?
• Score: 15.26161647903404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How do LLM agents come to both understand environments they act in and master tasks set within them? Through controlled experiments combining world-model training (next-state prediction) and policy training (reward maximization), we investigate this question. We dissect the resulting models through their additive parameter updates. Geometrically, we find effective world-model updates are low-rank and share an input-feature subspace with policy updates while writing to nearly orthogonal output directions, whether trained separately or sequentially. However, we find that, in projection interventions, the sequential update induces more robustness than separate policy RL when removing the world model's leading input directions, suggesting that it has learned alternative input pathways. Behaviorally, we find the sequentially trained agent explores a wider range of states and actions. Based on this, we ask: does policy training preserve world knowledge as well as it could? We probe this with training-free merging built on the geometrically motivated input basis plus an online world-model loss during policy RL, and show both improve over the untreated baseline. Our findings suggest world knowledge and task-directed ability can be learned in geometrically complementary forms, and that future post-training pipelines should consider how best to engineer the interface between them.

Related papers

• EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning [57.312638079204255]
  Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments.<n>We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal.
  arXiv  Detail & Related papers  (2026-08-06T15:54:36Z)
• When Does Continual Learning Require Learning [68.55508958298942]
  We show that continual learning is not a single capability.<n>Different patterns of environmental change require fundamentally different update behaviors.<n>We hope that understanding where each method succeeds and fails will guide the design of stronger continual learning systems.
  arXiv  Detail & Related papers  (2026-07-08T18:27:40Z)
• AIPO: Learning to Reason from Active Interaction [54.10819421625103]
  AIPO enables the policy model to proactively consult three functional collaborative agents when encountering reasoning bottlenecks.<n>AIPO consistently improves reasoning performance, generalizes robustly across different policy models and RLVR algorithms, and effectively expands the reasoning capability boundary of the policy model.
  arXiv  Detail & Related papers  (2026-05-08T19:06:55Z)
• Model Predictive Control with Differentiable World Models for Offline Reinforcement Learning [17.293286354647716]
  We introduce an inference time adaptation framework inspired by model predictive control (MPC)<n>We show that exploiting inference-time information to optimize the policy parameters yields consistent gains over strong offline RL baselines.
  arXiv  Detail & Related papers  (2026-03-23T18:05:29Z)
• Continual Reinforcement Learning by Planning with Online World Models [29.730338886371776]
  Continual reinforcement learning (CRL) refers to a naturalistic setting where an agent needs to endlessly evolve, by trial and error, to solve multiple tasks that are presented sequentially.<n>One of the largest obstacles to CRL is that the agent may forget how to solve previous tasks when learning a new task, known as catastrophic forgetting.<n>We propose to address this challenge by planning with online world models. Specifically, we learn a Follow-The-Leader shallow model online to capture the world dynamics, in which we plan using model predictive control to solve a set of tasks specified by any reward functions.
  arXiv  Detail & Related papers  (2025-07-12T07:52:31Z)
• LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models [42.53342297631436]
  Policy exploration is critical in reinforcement learning (RL), where existing approaches include greedy, Gaussian process, etc.<n>We design LLM-Explorer to adaptively generate task-specific exploration strategies with large language models (LLMs)<n>Our design is a plug-in module compatible with various widely applied RL algorithms, including the DQN series, DDPG, TD3, and any possible variants developed based on them.
  arXiv  Detail & Related papers  (2025-05-21T09:24:23Z)
• Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Models [19.05224410249602]
  We propose a novel approach for offline reinforcement learning with closed-loop policy evaluation and world-model adaptation. We analyzed the performance of the proposed method and provided an upper bound on the return gap between our method and the real environment under an optimal policy.
  arXiv  Detail & Related papers  (2024-05-30T09:34:31Z)
• How Can LLM Guide RL? A Value-Based Approach [68.55316627400683]
  Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. Recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities. We develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning.
  arXiv  Detail & Related papers  (2024-02-25T20:07:13Z)
• Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
  We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
  arXiv  Detail & Related papers  (2023-06-06T02:24:41Z)
• Domain Generalization for Robust Model-Based Offline Reinforcement Learning [5.653790804686631]
  Existing offline reinforcement learning algorithms assume that training data is either generated by a known policy, or of entirely unknown origin. We consider multi-demonstrator offline RL, a middle ground where we know which demonstrators generated each dataset, but make no assumptions about the underlying policies of the demonstrators. We propose Domain-Invariant Model-based Offline RL (DIMORL), where we apply Risk Extrapolation (REx) to the process of learning dynamics and rewards models.
  arXiv  Detail & Related papers  (2022-11-27T13:37:49Z)
• A Unified Framework for Alternating Offline Model Training and Policy Learning [62.19209005400561]
  In offline model-based reinforcement learning, we learn a dynamic model from historically collected data, and utilize the learned model and fixed datasets for policy learning. We develop an iterative offline MBRL framework, where we maximize a lower bound of the true expected return. With the proposed unified model-policy learning framework, we achieve competitive performance on a wide range of continuous-control offline reinforcement learning datasets.
  arXiv  Detail & Related papers  (2022-10-12T04:58:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.