UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning

• URL: http://arxiv.org/abs/2601.09215v1
• Date: Wed, 14 Jan 2026 06:42:01 GMT
• Title: UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning
• Authors: Feng Zhang, Shijia Li, Chunmao Zhang, Zhanyu Ma, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He, Jingwen Xu, Han Liu,
• Abstract summary: We propose UserLM-R1, a novel user language model with reasoning capability.<n>We first construct comprehensive user profiles with both static roles and dynamic scenario-specific goals for adaptation to diverse scenarios.<n>Then, we propose a goal-driven decision-making policy to generate high-quality rationales before producing responses.
• Score: 32.51053667574764
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: User simulators serve as the critical interactive environment for agent post-training, and an ideal user simulator generalizes across domains and proactively engages in negotiation by challenging or bargaining. However, current methods exhibit two issues. They rely on static and context-unaware profiles, necessitating extensive manual redesign for new scenarios, thus limiting generalizability. Moreover, they neglect human strategic thinking, leading to vulnerability to agent manipulation. To address these issues, we propose UserLM-R1, a novel user language model with reasoning capability. Specifically, we first construct comprehensive user profiles with both static roles and dynamic scenario-specific goals for adaptation to diverse scenarios. Then, we propose a goal-driven decision-making policy to generate high-quality rationales before producing responses, and further refine the reasoning and improve strategic capabilities with supervised fine-tuning and multi-reward reinforcement learning. Extensive experimental results demonstrate that UserLM-R1 outperforms competitive baselines, particularly on the more challenging adversarial set.

Related papers

• AI-Salesman: Towards Reliable Large Language Model Driven Telemarketing [79.0112532518727]
  We release TeleSalesCorpus, the first real-world-grounded dialogue dataset for this domain.<n>We then propose AI-Salesman, a novel framework featuring a dual-stage architecture.<n>We show that our proposed AI-Salesman significantly outperforms baseline models in both automatic metrics and comprehensive human evaluations.
  arXiv  Detail & Related papers  (2025-11-15T09:44:42Z)
• Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting [92.57796055887995]
  We introduce ECHO, a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents.<n> ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts.<n>We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation.
  arXiv  Detail & Related papers  (2025-10-11T18:11:09Z)
• UserRL: Training Interactive User-Centric Agent via Reinforcement Learning [104.63494870852894]
  Reinforcement learning (RL) has shown promise in training agentic models that engage in dynamic, multi-turn interactions.<n>We propose UserRL, a unified framework for training and evaluating user-centric abilities through standardized gym environments.
  arXiv  Detail & Related papers  (2025-09-24T03:33:20Z)
• CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards [53.36917093757101]
  Role-Playing Language Agents (RPLAs) have emerged as a significant application direction for Large Language Models (LLMs)<n>We introduce textbfCogDual, a novel RPLA adopting a textitcognize-then-respond reasoning paradigm.<n>By jointly modeling external situational awareness and internal self-awareness, CogDual generates responses with improved character consistency and contextual alignment.
  arXiv  Detail & Related papers  (2025-07-23T02:26:33Z)
• Reasoning LLMs for User-Aware Multimodal Conversational Agents [3.533721662684487]
  Personalization in social robotics is critical for fostering effective human-robot interactions.<n>This paper proposes a novel framework called USER-LLM R1 for a user-aware conversational agent.<n>Our approach integrates chain-of-thought (CoT) reasoning models to iteratively infer user preferences and vision-language models.
  arXiv  Detail & Related papers  (2025-04-02T13:00:17Z)
• Strategize Globally, Adapt Locally: A Multi-Turn Red Teaming Agent with Dual-Level Learning [39.931442440365444]
  AlgName is a novel red-teaming agent that emulates sophisticated human attackers through complementary learning dimensions.<n>AlgName enables the agent to identify new jailbreak tactics, develop a goal-based tactic selection framework, and refine prompt formulations for selected tactics.<n> Empirical evaluations on JailbreakBench demonstrate our framework's superior performance, achieving over 90% attack success rates against GPT-3.5-Turbo and Llama-3.1-70B within 5 conversation turns.
  arXiv  Detail & Related papers  (2025-04-02T01:06:19Z)
• Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles [37.43150003866563]
  We introduce the User Simulator with Implicit Profiles (USP), a framework that infers implicit user profiles from human-machine interactions to simulate personalized and realistic dialogues.<n>USP outperforms strong baselines in terms of authenticity and diversity while maintaining comparable consistency.
  arXiv  Detail & Related papers  (2025-02-26T09:26:54Z)
• Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalization [3.1944843830667766]
  Large language models (LLMs) have revolutionized how we interact with technology, but their personalization to individual user preferences remains a significant challenge. We present Adaptive Self-Supervised Learning Strategies (ASLS), which utilize self-supervised learning techniques to personalize LLMs dynamically.
  arXiv  Detail & Related papers  (2024-09-25T14:35:06Z)
• HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments [93.94020724735199]
  HAZARD consists of three unexpected disaster scenarios, including fire, flood, and wind. This benchmark enables us to evaluate autonomous agents' decision-making capabilities across various pipelines.
  arXiv  Detail & Related papers  (2024-01-23T18:59:43Z)
• Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition [64.06167416127386]
  We propose Multi-Agent Dialog Policy Learning, which regards both the system and the user as the dialog agents. Two agents interact with each other and are jointly learned simultaneously. Results show that our method can successfully build a system policy and a user policy simultaneously.
  arXiv  Detail & Related papers  (2020-04-08T04:51:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.