Aligning Language Models from User Interactions

• URL: http://arxiv.org/abs/2603.12273v1
• Date: Wed, 18 Feb 2026 16:31:35 GMT
• Title: Aligning Language Models from User Interactions
• Abstract summary: We propose a principled and scalable method for learning directly from user interactions through self-distillation.<n>We show that training on real-world user conversations from WildChat improves language models across standard alignment and instruction-following benchmarks.
• Score: 41.288783114966996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-turn user interactions are among the most abundant data produced by language models, yet we lack effective methods to learn from them. While typically discarded, these interactions often contain useful information: follow-up user messages may indicate that a response was incorrect, failed to follow an instruction, or did not align with the user's preferences. Importantly, language models are already able to make use of this information in context. After observing a user's follow-up, the same model is often able to revise its behavior. We leverage this ability to propose a principled and scalable method for learning directly from user interactions through self-distillation. By conditioning the model on the user's follow-up message and comparing the resulting token distribution with the original policy, we obtain a target for updating the policy that captures how the model's behavior changes in hindsight. We then distill this hindsight distribution back into the current policy. Remarkably, we show that training on real-world user conversations from WildChat improves language models across standard alignment and instruction-following benchmarks, without regressing other capabilities. The same mechanism enables personalization, allowing models to continually adapt to individual users through interaction without explicit feedback. Our results demonstrate that raw user interactions that arise naturally during deployment enable alignment, personalization, and continual adaptation.

Related papers

• Learning Dynamic User Personas from Implicit Interaction Streams via Iterative Refinement [4.027292457517535]
  We propose IRIS, a framework that learns dynamic user personas directly from implicit interaction streams.<n>IRIS produces stable personas and distinguishes individual users while revealing limitations of memory-only approaches.<n>We validate IRIS on anonymized real-world Reddit r/AmItheAsshole (AITA) data, with personas built solely from each author's historical interactions.
  arXiv  Detail & Related papers  (2026-07-29T04:56:14Z)
• Preference-Aware Rubric Learning for Personalized Evaluation [59.539429430690156]
  Existing evaluation methods fail to capture user-specific preferences embedded in long-term interaction histories.<n>We propose Personalized Evaluation as Learning, a paradigm that formulates personalized evaluation as a learning problem rather than a static judgment.<n>Experiments show that PARL consistently induces high-fidelity rubrics that reliably identify user-aligned responses and generalize across users.
  arXiv  Detail & Related papers  (2026-05-29T17:00:55Z)
• Can We Predict the Next Question? A Collaborative Filtering Approach to Modeling User Behavior [16.241726074740082]
  Large language models (LLMs) have excelled in language understanding and generation, powering advanced dialogue and recommendation systems.<n>We propose a Collaborative Filtering-enhanced Question Prediction framework to bridge the gap between language modeling and behavioral sequence modeling.
  arXiv  Detail & Related papers  (2025-11-17T04:01:20Z)
• The Era of Real-World Human Interaction: RL from User Conversations [45.2392745984914]
  We introduce Reinforcement Learning from Human Interaction (RLHI), a paradigm that learns directly from in-the-wild user conversations.<n>We develop two complementary methods: (1) RLHI with User-Guided Rewrites, which revises unsatisfactory model outputs based on users' natural-language follow-up responses, and (2) RLHI with User-Based Rewards, which learns via a reward model conditioned on knowledge of the user's long-term interaction history.
  arXiv  Detail & Related papers  (2025-09-29T17:50:31Z)
• Learning to summarize user information for personalized reinforcement learning from human feedback [19.859785715555013]
  Preference Learning Using Summarization (PLUS) uses reinforcement learning to learn to produce text-based summaries of each user's preferences.<n>Both the user-summarization model and reward model are trained simultaneously, creating an online co-adaptation loop.<n>We show that PLUS capture diverse aspects of user preferences, achieving a 11-77% improvement in reward model accuracy.
  arXiv  Detail & Related papers  (2025-07-17T23:48:51Z)
• Human Learning by Model Feedback: The Dynamics of Iterative Prompting with Midjourney [28.39697076030535]
  This paper analyzes the dynamics of the user prompts along such iterations. We show that prompts predictably converge toward specific traits along these iterations. The possibility that users adapt to the model's preference raises concerns about reusing user data for further training.
  arXiv  Detail & Related papers  (2023-11-20T19:28:52Z)
• RecExplainer: Aligning Large Language Models for Explaining Recommendation Models [50.74181089742969]
  Large language models (LLMs) have demonstrated remarkable intelligence in understanding, reasoning, and instruction following. This paper presents the initial exploration of using LLMs as surrogate models to explain black-box recommender models. To facilitate an effective alignment, we introduce three methods: behavior alignment, intention alignment, and hybrid alignment.
  arXiv  Detail & Related papers  (2023-11-18T03:05:43Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Chain of Hindsight Aligns Language Models with Feedback [62.68665658130472]
  We propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors.
  arXiv  Detail & Related papers  (2023-02-06T10:28:16Z)
• Latent User Intent Modeling for Sequential Recommenders [92.66888409973495]
  Sequential recommender models learn to predict the next items a user is likely to interact with based on his/her interaction history on the platform. Most sequential recommenders however lack a higher-level understanding of user intents, which often drive user behaviors online. Intent modeling is thus critical for understanding users and optimizing long-term user experience.
  arXiv  Detail & Related papers  (2022-11-17T19:00:24Z)
• Modeling Behaviour to Predict User State: Self-Reports as Ground Truth [28.327411414505306]
  Methods that detect user states such as emotions are useful for interactive systems. We argue for model-based approaches that are trained on user behaviour and self-reported user state as ground truths.
  arXiv  Detail & Related papers  (2020-07-28T20:09:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.