Learning to Simulate Human Dialogue

• URL: http://arxiv.org/abs/2601.04436v1
• Date: Wed, 07 Jan 2026 22:51:31 GMT
• Title: Learning to Simulate Human Dialogue
• Authors: Kanishk Gandhi, Agam Bhatia, Noah D. Goodman,
• Abstract summary: Next-turn dialogue prediction: given a conversation, predict the next utterance produced by a person.<n>We compare learning approaches along two dimensions: whether the model is allowed to think before responding, and how learning is rewarded.<n>We find that thinking helps primarily when trained with a distribution-matching objective grounded in real human dialogue.
• Score: 35.88351482220924
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To predict what someone will say is to model how they think. We study this through next-turn dialogue prediction: given a conversation, predict the next utterance produced by a person. We compare learning approaches along two dimensions: (1) whether the model is allowed to think before responding, and (2) how learning is rewarded either through an LLM-as-a-judge that scores semantic similarity and information completeness relative to the ground-truth response, or by directly maximizing the log-probability of the true human dialogue. We find that optimizing for judge-based rewards indeed increases judge scores throughout training, however it decreases the likelihood assigned to ground truth human responses and decreases the win rate when human judges choose the most human-like response among a real and synthetic option. This failure is amplified when the model is allowed to think before answering. In contrast, by directly maximizing the log-probability of observed human responses, the model learns to better predict what people actually say, improving on both log-probability and win rate evaluations. Treating chain-of-thought as a latent variable, we derive a lower bound on the log-probability. Optimizing this objective yields the best results on all our evaluations. These results suggest that thinking helps primarily when trained with a distribution-matching objective grounded in real human dialogue, and that scaling this approach to broader conversational data may produce models with a more nuanced understanding of human behavior.

Related papers

• Leveraging Implicit Feedback from Deployment Data in Dialogue [83.02878726357523]
  We study improving social conversational agents by learning from natural dialogue between users and a deployed model. We leverage signals like user response length, sentiment and reaction of the future human utterances in the collected dialogue episodes.
  arXiv  Detail & Related papers  (2023-07-26T11:34:53Z)
• Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems [82.92678837778358]
  preference-based methods have demonstrated substantial success in empirical applications such as InstructGPT. We show how human bias and uncertainty in feedback modelings can affect the theoretical guarantees of these approaches.
  arXiv  Detail & Related papers  (2023-07-24T17:50:24Z)
• Chain of Hindsight Aligns Language Models with Feedback [62.68665658130472]
  We propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors.
  arXiv  Detail & Related papers  (2023-02-06T10:28:16Z)
• The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types [38.37216644899506]
  We argue that grounding the rationality coefficient in real data for each feedback type has a significant positive effect on reward learning. We find that when learning from a single feedback type, overestimating human rationality can have dire effects on reward accuracy and regret.
  arXiv  Detail & Related papers  (2022-08-23T02:19:10Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• Dialogue Response Ranking Training with Large-Scale Human Feedback Data [52.12342165926226]
  We leverage social media feedback data to build a large-scale training dataset for feedback prediction. We trained DialogRPT, a set of GPT-2 based models on 133M pairs of human feedback data. Our ranker outperforms the conventional dialog perplexity baseline with a large margin on predicting Reddit feedback.
  arXiv  Detail & Related papers  (2020-09-15T10:50:05Z)
• Improving Factual Consistency Between a Response and Persona Facts [64.30785349238619]
  Neural models for response generation produce responses that are semantically plausible but not necessarily factually consistent with facts describing the speaker's persona. We propose to fine-tune these models by reinforcement learning and an efficient reward function that explicitly captures the consistency between a response and persona facts as well as semantic plausibility.
  arXiv  Detail & Related papers  (2020-04-30T18:08:22Z)
• "Wait, I'm Still Talking!" Predicting the Dialogue Interaction Behavior Using Imagine-Then-Arbitrate Model [24.560203199376478]
  In real human-human conversations, human often sequentially sends several short messages for readability instead of a long message in one turn. We propose a novel Imagine-then-Arbitrate (ITA) neural dialogue model to help the agent decide whether to wait or to make a response directly.
  arXiv  Detail & Related papers  (2020-02-22T04:05:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.