In-Context Learning User Simulators for Task-Oriented Dialog Systems

• URL: http://arxiv.org/abs/2306.00774v1
• Date: Thu, 1 Jun 2023 15:06:11 GMT
• Title: In-Context Learning User Simulators for Task-Oriented Dialog Systems
• Authors: Silvia Terragni, Modestas Filipavicius, Nghia Khau, Bruna Guedes, Andr\'e Manso, Roland Mathis
• Abstract summary: This paper presents a novel application of large language models in user simulation for task-oriented dialog systems. By harnessing the power of these models, the proposed approach generates diverse utterances based on user goals and limited dialog examples.
• Score: 1.7086737326992172
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a novel application of large language models in user simulation for task-oriented dialog systems, specifically focusing on an in-context learning approach. By harnessing the power of these models, the proposed approach generates diverse utterances based on user goals and limited dialog examples. Unlike traditional simulators, this method eliminates the need for labor-intensive rule definition or extensive annotated data, making it more efficient and accessible. Additionally, an error analysis of the interaction between the user simulator and dialog system uncovers common mistakes, providing valuable insights into areas that require improvement. Our implementation is available at https://github.com/telepathylabsai/prompt-based-user-simulator.

Related papers

• Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems [2.788542465279969]
  This paper introduces DAUS, a Domain-Aware User Simulator. We fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment.
  arXiv  Detail & Related papers  (2024-02-20T20:57:47Z)
• Dialogue-based generation of self-driving simulation scenarios using Large Language Models [14.86435467709869]
  Simulation is an invaluable tool for developing and evaluating controllers for self-driving cars. Current simulation frameworks are driven by highly-specialist domain specific languages. There is often a gap between a concise English utterance and the executable code that captures the user's intent.
  arXiv  Detail & Related papers  (2023-10-26T13:07:01Z)
• Improving Conversational Recommendation Systems via Counterfactual Data Simulation [73.4526400381668]
  Conversational recommender systems (CRSs) aim to provide recommendation services via natural language conversations. Existing CRS approaches often suffer from the issue of insufficient training due to the scarcity of training data. We propose a CounterFactual data simulation approach for CRS, named CFCRS, to alleviate the issue of data scarcity in CRSs.
  arXiv  Detail & Related papers  (2023-06-05T12:48:56Z)
• Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Tracking [57.92608483099916]
  Large pre-trained language models (PLMs) have shown impressive unaided performance across many NLP tasks. For more complex tasks such as dialogue state tracking (DST), designing prompts that reliably convey the desired intent is nontrivial. We introduce a saliency model to limit dialogue text length, allowing us to include more exemplars per query.
  arXiv  Detail & Related papers  (2023-02-12T15:05:10Z)
• Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems [80.77917437785773]
  Task-oriented dialogue systems ( TDSs) are assessed mainly in an offline setting or through human evaluation. We propose a metaphorical user simulator for end-to-end TDS evaluation, where we define a simulator to be metaphorical if it simulates user's analogical thinking in interactions with systems. We also propose a tester-based evaluation framework to generate variants, i.e., dialogue systems with different capabilities.
  arXiv  Detail & Related papers  (2022-04-02T05:11:03Z)
• Alexa Conversations: An Extensible Data-driven Approach for Building Task-oriented Dialogue Systems [21.98135285833616]
  Traditional goal-oriented dialogue systems rely on various components such as natural language understanding, dialogue state tracking, policy learning and response generation. We present a new approach for building goal-oriented dialogue systems that is scalable, as well as data efficient.
  arXiv  Detail & Related papers  (2021-04-19T07:09:27Z)
• Dialog Simulation with Realistic Variations for Training Goal-Oriented Conversational Systems [14.206866126142002]
  Goal-oriented dialog systems enable users to complete specific goals like requesting information about a movie or booking a ticket. We propose an approach for automatically creating a large corpus of annotated dialogs from a few thoroughly annotated sample dialogs and the dialog schema. We achieve 18? 50% relative accuracy on a held-out test set compared to a baseline dialog generation approach.
  arXiv  Detail & Related papers  (2020-11-16T19:39:15Z)
• SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching [81.45928589522032]
  We parameterize modular task-oriented dialog systems using a Transformer-based auto-regressive language model. We pre-train, on heterogeneous dialog corpora, a task-grounded response generation model. Experiments show that SOLOIST creates new state-of-the-art on well-studied task-oriented dialog benchmarks.
  arXiv  Detail & Related papers  (2020-05-11T17:58:34Z)
• Conversation Learner -- A Machine Teaching Tool for Building Dialog Managers for Task-Oriented Dialog Systems [57.082447660944965]
  Conversation Learner is a machine teaching tool for building dialog managers. It enables dialog authors to create a dialog flow using familiar tools, converting the dialog flow into a parametric model. It allows dialog authors to improve the dialog manager over time by leveraging user-system dialog logs as training data.
  arXiv  Detail & Related papers  (2020-04-09T00:10:54Z)
• Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition [64.06167416127386]
  We propose Multi-Agent Dialog Policy Learning, which regards both the system and the user as the dialog agents. Two agents interact with each other and are jointly learned simultaneously. Results show that our method can successfully build a system policy and a user policy simultaneously.
  arXiv  Detail & Related papers  (2020-04-08T04:51:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.