World Model-Guided Reinforcement Learning via Counterfactual User Engagement Simulation

• URL: http://arxiv.org/abs/2609.01067v1
• Date: Tue, 01 Sep 2026 10:59:41 GMT
• Title: World Model-Guided Reinforcement Learning via Counterfactual User Engagement Simulation
• Abstract summary: Reinforcement learning for user-centric agents is limited by the cost, latency, and risk of collecting online feedback.<n>We propose World Model-Guided Reinforcement Learning via counterfactual user engagement simulation (WMG-RL)<n>We show that UEWM provides reliable and transferable reward signals across domains.
• Score: 43.27233360158595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning for user-centric agents is limited by the cost, latency, and risk of collecting online feedback, as well as by the lack of counterfactual comparisons under the same user state. In this paper, we propose World Model-Guided Reinforcement Learning via counterfactual user engagement simulation (WMG-RL), a framework in which a frozen user simulator provides reward supervision before real user exposure. Motivated by language world models, we instantiate the simulator as a User Engagement World Model (UEWM), which treats a recommended item as the agent action and the user's heterogeneous feedback as the environment observation. Rather than learning one fixed environment transition, UEWM learns to infer user-specific dynamics from engagement history and apply them to candidate items. In WMG-RL, a downstream policy proposes multiple candidate items for the same history; UEWM predicts the corresponding engagement feedback in parallel; and the simulated feedback is converted into dense rewards for policy optimization. Experiments show that UEWM provides reliable and transferable reward signals across domains, and that WMG-RL enables a compact 1.7B student policy to match or surpass much larger LLMs on downstream recommendation tasks.

Related papers

• User Simulator-Guided Multi-Turn Preference Optimization for Reasoning LLM-based Conversational Recommendation [5.313432318455876]
  We propose a user simulator-guided multi-turn preference optimization conversational recommendation framework.<n>To align simulator-generated feedback with true user preferences in the absence of explicit labels, we enhance feedback quality via multi-task supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-04-04T09:52:30Z)
• Mind the Sim2Real Gap in User Simulation for Agentic Tasks [101.69142591891234]
  We formalize the Sim2Real gap in user simulation and present the first study running the full $$-bench protocol with real humans.<n> Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity.<n>These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle.
  arXiv  Detail & Related papers  (2026-03-11T19:12:31Z)
• AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluation [0.7031557790463293]
  We introduce Align, a framework that learns world-model-driven agents from human interactions.<n>We generate counterfactual trajectories around demonstrations and prompt the LLM to compare its decisions with human choices, identify suboptimal actions, and extract lessons.
  arXiv  Detail & Related papers  (2026-01-02T03:01:33Z)
• UserRL: Training Interactive User-Centric Agent via Reinforcement Learning [104.63494870852894]
  Reinforcement learning (RL) has shown promise in training agentic models that engage in dynamic, multi-turn interactions.<n>We propose UserRL, a unified framework for training and evaluating user-centric abilities through standardized gym environments.
  arXiv  Detail & Related papers  (2025-09-24T03:33:20Z)
• RecoWorld: Building Simulated Environments for Agentic Recommender Systems [55.979427290369216]
  We present RecoWorld, a blueprint for building simulated environments tailored to agentic recommender systems.<n>A user simulator reviews recommended items, updates its mindset, and when sensing potential user disengagement, generates reflective instructions.<n>The agentic recommender adapts its recommendations by incorporating these user instructions and reasoning traces, creating a dynamic feedback loop.
  arXiv  Detail & Related papers  (2025-09-12T16:44:34Z)
• RecGPT Technical Report [57.84251629878726]
  We propose RecGPT, a next-generation framework that places user intent at the center of the recommendation pipeline.<n> RecGPT integrates large language models into key stages of user interest mining, item retrieval, and explanation generation.<n>Online experiments demonstrate that RecGPT achieves consistent performance gains across stakeholders.
  arXiv  Detail & Related papers  (2025-07-30T17:55:06Z)
• Large Language Model Empowered Recommendation Meets All-domain Continual Pre-Training [60.38082979765664]
  CPRec is an All-domain Continual Pre-Training framework for Recommendation.<n>It holistically align LLMs with universal user behaviors through the continual pre-training paradigm.<n>We conduct experiments on five real-world datasets from two distinct platforms.
  arXiv  Detail & Related papers  (2025-04-11T20:01:25Z)
• Lusifer: LLM-based User SImulated Feedback Environment for online Recommender systems [0.0]
  Reinforcement learning (RL) recommender systems often rely on static datasets that fail to capture the fluid, ever changing nature of user preferences in real-world scenarios.<n>We introduce Lusifer, an LLM-based simulation environment designed to generate dynamic, realistic user feedback for RL-based recommender training.
  arXiv  Detail & Related papers  (2024-05-22T05:43:15Z)
• A LLM-based Controllable, Scalable, Human-Involved User Simulator Framework for Conversational Recommender Systems [14.646529557978512]
  Conversational Recommender System (CRS) leverages real-time feedback from users to dynamically model their preferences. Large Language Models (LLMs) has marked the onset of a new epoch in computational capabilities. We introduce a Controllable, scalable, and human-Involved (CSHI) simulator framework that manages the behavior of user simulators.
  arXiv  Detail & Related papers  (2024-05-13T03:02:56Z)
• User Behavior Simulation with Large Language Model based Agents [116.74368915420065]
  We propose an LLM-based agent framework and design a sandbox environment to simulate real user behaviors. Based on extensive experiments, we find that the simulated behaviors of our method are very close to the ones of real humans.
  arXiv  Detail & Related papers  (2023-06-05T02:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.