Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping

• URL: http://arxiv.org/abs/2510.07230v2
• Date: Sat, 18 Oct 2025 04:00:48 GMT
• Title: Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping
• Authors: Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Jing Huang, Dakuo Wang,
• Abstract summary: We introduce Customer-R1, an RL-based method for personalized, step-wise user behavior simulation in online shopping environments.<n>Our policy is conditioned on an explicit persona, and we optimize next-step rationale and action generation via action correctness reward signals.
• Score: 27.626024821315486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Simulating step-wise human behavior with Large Language Models (LLMs) has become an emerging research direction, enabling applications in various practical domains. While prior methods, including prompting, supervised fine-tuning (SFT), and reinforcement learning (RL), have shown promise in modeling step-wise behavior, they primarily learn a population-level policy without conditioning on a user's persona, yielding generic rather than personalized simulations. In this work, we pose a critical question: how can LLM agents better simulate personalized user behavior? We introduce Customer-R1, an RL-based method for personalized, step-wise user behavior simulation in online shopping environments. Our policy is conditioned on an explicit persona, and we optimize next-step rationale and action generation via action correctness reward signals. Experiments on the OPeRA dataset emonstrate that Customer-R1 not only significantly outperforms prompting and SFT-based baselines in next-action prediction tasks, but also better matches users' action distribution, indicating higher fidelity in personalized behavior simulation.

Related papers

• SPACeR: Self-Play Anchoring with Centralized Reference Models [50.55045557371374]
  Sim agent policies are realistic, human-like, fast, and scalable in multi-agent settings.<n>Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data.<n>We propose SPACeR, a framework that leverages a pretrained tokenized autoregressive motion model as a central reference policy.
  arXiv  Detail & Related papers  (2025-10-20T19:53:02Z)
• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors [58.87134689752605]
  We introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation.<n>We show that even the best LLMs today have limited simulation ability (score: 40.80/100), performance scales log-linearly with model size.<n>We demonstrate that simulation ability correlates most strongly with deep, knowledge-intensive reasoning.
  arXiv  Detail & Related papers  (2025-10-20T13:14:38Z)
• Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning [35.83999932977034]
  We propose a novel R1-style reinforcement fine-tuning paradigm tailored for next-token prediction models to better align agent behavior with human preferences and evaluation metrics.<n>Our approach introduces a metric-oriented policy optimization algorithm to improve distribution alignment and an iterative "SFT-RFT-SFT" training strategy that alternates between Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT)<n>The results on the Open Sim Agents Challenge showcase that SMART-R1 achieves state-of-the-art performance with an overall realism meta score of 0.7858, ranking first on
  arXiv  Detail & Related papers  (2025-09-28T17:36:13Z)
• Implicit Behavioral Alignment of Language Agents in High-Stakes Crowd Simulations [3.0112218223206173]
  Language-driven generative agents have enabled social simulations with transformative uses, from interpersonal training to aiding global policy-making.<n>Recent studies indicate that generative agent behaviors often deviate from expert expectations and real-world data--a phenomenon we term the Behavior-Realism Gap.<n>We introduce a theoretical framework called Persona-Environment Behavioral Alignment (PEBA), formulated as a distribution matching problem grounded in Lewin's behavior equation.<n>We propose PersonaEvolve (PEvo), an LLM-based optimization algorithm that iteratively refines agent personas, implicitly aligning their collective behaviors with realistic expert benchmarks within a specified environmental context.
  arXiv  Detail & Related papers  (2025-09-19T22:35:13Z)
• Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning [27.226155951073064]
  Shop-R1 is a novel reinforcement learning framework aimed at enhancing the reasoning ability of Large Language Models (LLMs)<n>For rationale generation, we leverage internal model signals (e.g., logit distributions) to guide the reasoning process in a self-supervised manner.<n>For action prediction, we propose a hierarchical reward structure with difficulty-aware scaling to prevent reward hacking.
  arXiv  Detail & Related papers  (2025-07-23T18:10:43Z)
• OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation [56.47029531207105]
  OPERA is the first public dataset that comprehensively captures user personas, browser observations, fine-grained web actions, and self-reported just-in-time rationales.<n>We establish the first benchmark to evaluate how well current LLMs can predict a specific user's next action and rationale.
  arXiv  Detail & Related papers  (2025-06-05T21:37:49Z)
• Prompting is Not All You Need! Evaluating LLM Agent Simulation Methodologies with Real-World Online Customer Behavior Data [62.61900377170456]
  We focus on evaluating LLM's objective accuracy'' rather than the subjective believability'' in simulating human behavior.<n>We present the first comprehensive evaluation of state-of-the-art LLMs on the task of web shopping action generation.
  arXiv  Detail & Related papers  (2025-03-26T17:33:27Z)
• Large Language Model driven Policy Exploration for Recommender Systems [50.70228564385797]
  offline RL policies trained on static user data are vulnerable to distribution shift when deployed in dynamic online environments.<n>Online RL-based RS also face challenges in production deployment due to the risks of exposing users to untrained or unstable policies.<n>Large Language Models (LLMs) offer a promising solution to mimic user objectives and preferences for pre-training policies offline.<n>We propose an Interaction-Augmented Learned Policy (iALP) that utilizes user preferences distilled from an LLM.
  arXiv  Detail & Related papers  (2025-01-23T16:37:44Z)
• SALMON: Self-Alignment with Instructable Reward Models [80.83323636730341]
  This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision. We develop an AI assistant named Dromedary-2 with only 6 exemplars for in-context learning and 31 human-defined principles.
  arXiv  Detail & Related papers  (2023-10-09T17:56:53Z)
• User Behavior Simulation with Large Language Model based Agents [116.74368915420065]
  We propose an LLM-based agent framework and design a sandbox environment to simulate real user behaviors. Based on extensive experiments, we find that the simulated behaviors of our method are very close to the ones of real humans.
  arXiv  Detail & Related papers  (2023-06-05T02:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.