The Era of Real-World Human Interaction: RL from User Conversations

• URL: http://arxiv.org/abs/2509.25137v1
• Date: Mon, 29 Sep 2025 17:50:31 GMT
• Title: The Era of Real-World Human Interaction: RL from User Conversations
• Abstract summary: We introduce Reinforcement Learning from Human Interaction (RLHI), a paradigm that learns directly from in-the-wild user conversations.<n>We develop two complementary methods: (1) RLHI with User-Guided Rewrites, which revises unsatisfactory model outputs based on users' natural-language follow-up responses, and (2) RLHI with User-Based Rewards, which learns via a reward model conditioned on knowledge of the user's long-term interaction history.
• Score: 45.2392745984914
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We posit that to achieve continual model improvement and multifaceted alignment, future models must learn from natural human interaction. Current conversational models are aligned using pre-annotated, expert-generated human feedback. In this work, we introduce Reinforcement Learning from Human Interaction (RLHI), a paradigm that learns directly from in-the-wild user conversations. We develop two complementary methods: (1) RLHI with User-Guided Rewrites, which revises unsatisfactory model outputs based on users' natural-language follow-up responses, (2) RLHI with User-Based Rewards, which learns via a reward model conditioned on knowledge of the user's long-term interaction history (termed persona). Together, these methods link long-term user personas to turn-level preferences via persona-conditioned preference optimization. Trained on conversations derived from WildChat, both RLHI variants outperform strong baselines in personalization and instruction-following, and similar feedback enhances performance on reasoning benchmarks. These results suggest organic human interaction offers scalable, effective supervision for personalized alignment.

Related papers

• Interplay: Training Independent Simulators for Reference-Free Conversational Recommendation [29.520574514713402]
  We propose a reference-free simulation framework that trains two independent large language models (LLMs)<n>These models interact in real-time without access to predetermined target items, but preference summaries and target attributes.<n>This approach produces more realistic and diverse conversations that closely mirror authentic human-AI interactions.
  arXiv  Detail & Related papers  (2026-03-19T07:33:50Z)
• Aligning Language Models from User Interactions [41.288783114966996]
  We propose a principled and scalable method for learning directly from user interactions through self-distillation.<n>We show that training on real-world user conversations from WildChat improves language models across standard alignment and instruction-following benchmarks.
  arXiv  Detail & Related papers  (2026-02-18T16:31:35Z)
• Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning [66.52010873968383]
  We introduce a conversational agent that interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through reinforcement learning (RL) training.<n>The experimental results across four widely used conversational benchmarks demonstrate the effectiveness of our methods.
  arXiv  Detail & Related papers  (2026-01-19T14:55:54Z)
• Mind the Gap: Linguistic Divergence and Adaptation Strategies in Human-LLM Assistant vs. Human-Human Interactions [14.21024646209994]
  Large Language Models (LLMs) are increasingly deployed in customer-facing applications.<n>Our study shows significant differences in grammatical fluency, politeness, and lexical diversity in user language between the two settings.<n>To enhance robustness to post-launch communication style changes, we experimented with two strategies.
  arXiv  Detail & Related papers  (2025-10-03T00:45:37Z)
• Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment [35.68913976348608]
  We introduce the Reinforcement Learning for Personalized Alignment (RLPA) framework to iteratively infer and refine user profiles through dialogue.<n>We instantiate RLPA by fine-tuning Qwen-2.5-3B-Instruct, resulting in Qwen-RLPA, which achieves state-of-the-art performance in personalized dialogue.
  arXiv  Detail & Related papers  (2025-05-21T12:38:36Z)
• Search-Based Interaction For Conversation Recommendation via Generative Reward Model Based Simulated User [117.82681846559909]
  Conversational recommendation systems (CRSs) use multi-turn interaction to capture user preferences and provide personalized recommendations.<n>We propose a generative reward model based simulated user, named GRSU, for automatic interaction with CRSs.
  arXiv  Detail & Related papers  (2025-04-29T06:37:30Z)
• Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward [11.495697919066341]
  We propose leveraging a user model to incorporate a curiosity-based intrinsic reward into multi-turn RLHF.<n>This novel reward mechanism encourages the LLM agent to actively infer user traits by optimizing conversations to improve its user model's accuracy.<n>We demonstrate our method's effectiveness in two distinct domains: significantly improving personalization performance in a conversational recommendation task, and personalizing conversations for different learning styles in an educational setting.
  arXiv  Detail & Related papers  (2025-04-04T06:35:02Z)
• Reasoning LLMs for User-Aware Multimodal Conversational Agents [3.533721662684487]
  Personalization in social robotics is critical for fostering effective human-robot interactions.<n>This paper proposes a novel framework called USER-LLM R1 for a user-aware conversational agent.<n>Our approach integrates chain-of-thought (CoT) reasoning models to iteratively infer user preferences and vision-language models.
  arXiv  Detail & Related papers  (2025-04-02T13:00:17Z)
• Aligning Language Models Using Follow-up Likelihood as Reward Signal [40.388526412214276]
  We propose using the likelihood of follow-up utterances as rewards to differentiate preferred responses from less favored ones.<n>Our proposed reward mechanism, Follow-up Likelihood as Reward" (FLR), matches the performance of strong reward models trained on large-scale human or GPT-4 annotated data.
  arXiv  Detail & Related papers  (2024-09-20T23:47:25Z)
• Lusifer: LLM-based User SImulated Feedback Environment for online Recommender systems [0.0]
  Reinforcement learning (RL) recommender systems often rely on static datasets that fail to capture the fluid, ever changing nature of user preferences in real-world scenarios.<n>We introduce Lusifer, an LLM-based simulation environment designed to generate dynamic, realistic user feedback for RL-based recommender training.
  arXiv  Detail & Related papers  (2024-05-22T05:43:15Z)
• Personalized Language Modeling from Personalized Human Feedback [45.16986573937782]
  Personalized large language models (LLMs) are designed to tailor responses to individual user preferences.<n>We propose Personalized-RLHF, an efficient framework that utilizes a lightweight user model to capture individual user preferences.<n>We show that personalized LLMs trained using P-RLHF generate responses that are more closely aligned with individual user preferences.
  arXiv  Detail & Related papers  (2024-02-06T04:18:58Z)
• UltraFeedback: Boosting Language Models with Scaled AI Feedback [99.4633351133207]
  We present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models.
  arXiv  Detail & Related papers  (2023-10-02T17:40:01Z)
• Chain of Hindsight Aligns Language Models with Feedback [62.68665658130472]
  We propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors.
  arXiv  Detail & Related papers  (2023-02-06T10:28:16Z)
• Dialogue Response Ranking Training with Large-Scale Human Feedback Data [52.12342165926226]
  We leverage social media feedback data to build a large-scale training dataset for feedback prediction. We trained DialogRPT, a set of GPT-2 based models on 133M pairs of human feedback data. Our ranker outperforms the conventional dialog perplexity baseline with a large margin on predicting Reddit feedback.
  arXiv  Detail & Related papers  (2020-09-15T10:50:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.