LLM Roleplay: Simulating Human-Chatbot Interaction
- URL: http://arxiv.org/abs/2407.03974v1
- Date: Thu, 4 Jul 2024 14:49:46 GMT
- Title: LLM Roleplay: Simulating Human-Chatbot Interaction
- Authors: Hovhannes Tamoyan, Hendrik Schuff, Iryna Gurevych,
- Abstract summary: LLM-Roleplay is a goal-oriented, persona-based method to automatically generate diverse multi-turn dialogues simulating human-chatbot interaction.
We collect natural human-chatbot dialogues from different sociodemographic groups and conduct a human evaluation to compare real human-chatbot dialogues with our generated dialogues.
- Score: 52.03241266241294
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The development of chatbots requires collecting a large number of human-chatbot dialogues to reflect the breadth of users' sociodemographic backgrounds and conversational goals. However, the resource requirements to conduct the respective user studies can be prohibitively high and often only allow for a narrow analysis of specific dialogue goals and participant demographics. In this paper, we propose LLM-Roleplay: a goal-oriented, persona-based method to automatically generate diverse multi-turn dialogues simulating human-chatbot interaction. LLM-Roleplay can be applied to generate dialogues with any type of chatbot and uses large language models (LLMs) to play the role of textually described personas. To validate our method we collect natural human-chatbot dialogues from different sociodemographic groups and conduct a human evaluation to compare real human-chatbot dialogues with our generated dialogues. We compare the abilities of state-of-the-art LLMs in embodying personas and holding a conversation and find that our method can simulate human-chatbot dialogues with a high indistinguishability rate.
Related papers
- Stephanie: Step-by-Step Dialogues for Mimicking Human Interactions in Social Conversations [50.698517967337885]
We introduce a novel textbf-by-Step Dialogue Paradigm (Stephanie), designed to mimic the ongoing dynamic nature of human conversations.
By employing a dual learning strategy and a further-split post-editing method, we generated and utilized a high-quality step-by-step dialogue dataset.
Tailored automatic and human evaluations are conducted to assess its effectiveness compared to the traditional single-step dialogue paradigm.
arXiv Detail & Related papers (2024-07-04T17:59:41Z) - Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation [55.043492250775294]
We introduce a novel Face-to-Face spoken dialogue model.
It processes audio-visual speech from user input and generates audio-visual speech as the response.
We also introduce MultiDialog, the first large-scale multimodal spoken dialogue corpus.
arXiv Detail & Related papers (2024-06-12T04:48:36Z) - From Human-to-Human to Human-to-Bot Conversations in Software Engineering [3.1747517745997014]
We aim to understand the dynamics of conversations that occur during modern software development after the integration of AI and chatbots.
We compile existing conversation attributes with humans and NLU-based chatbots and adapt them to the context of software development.
We present similarities and differences between human-to-human and human-to-bot conversations.
We conclude that the recent conversation styles that we observe with LLM-chatbots can not replace conversations with humans.
arXiv Detail & Related papers (2024-05-21T12:04:55Z) - Ain't Misbehavin' -- Using LLMs to Generate Expressive Robot Behavior in
Conversations with the Tabletop Robot Haru [9.2526849536751]
We introduce a fully-automated conversation system that leverages large language models (LLMs) to generate robot responses with expressive behaviors.
We conduct a pilot study where volunteers chat with a social robot using our proposed system, and we analyze their feedback, conducting a rigorous error analysis of chat transcripts.
Most negative feedback was due to automatic speech recognition (ASR) errors which had limited impact on conversations.
arXiv Detail & Related papers (2024-02-18T12:35:52Z) - BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues [72.65163468440434]
This report provides a preliminary evaluation of existing large language models for human-style multi-turn chatting.
We prompt large language models (LLMs) to generate a full multi-turn dialogue based on the ChatSEED, utterance by utterance.
We find GPT-4 can generate human-style multi-turn dialogues with impressive quality, significantly outperforms its counterparts.
arXiv Detail & Related papers (2023-10-20T16:53:51Z) - PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator [39.40718009289621]
We propose a paradigm to simulate human behavior better and explore the benefits of incorporating more human-like questions in multi-turn conversations.
Specifically, we target human questions extracted from genuine human-machine conversations as a learning goal and provide a novel user simulator called Socratic'
Our results show our response model, PlatoLM', achieves SoTA performance among LLaMA-based 7B models in MT-Bench.
arXiv Detail & Related papers (2023-08-21T06:51:56Z) - PLACES: Prompting Language Models for Social Conversation Synthesis [103.94325597273316]
We use a small set of expert-written conversations as in-context examples to synthesize a social conversation dataset using prompting.
We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations.
arXiv Detail & Related papers (2023-02-07T05:48:16Z) - Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn
Chatbot Responding with Intention [55.77218465471519]
This paper proposes an innovative framework to train chatbots to possess human-like intentions.
Our framework included a guiding robot and an interlocutor model that plays the role of humans.
We examined our framework using three experimental setups and evaluate the guiding robot with four different metrics to demonstrated flexibility and performance advantages.
arXiv Detail & Related papers (2021-03-30T15:24:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.