Identifying Breakdowns in Conversational Recommender Systems using User Simulation

• URL: http://arxiv.org/abs/2405.14249v1
• Date: Thu, 23 May 2024 07:28:26 GMT
• Title: Identifying Breakdowns in Conversational Recommender Systems using User Simulation
• Authors: Nolwenn Bernard, Krisztian Balog,
• Abstract summary: We present a methodology to test conversational recommender systems with regards to conversational breakdowns. It involves examining conversations generated between the system and simulated users for a set of pre-defined breakdown types. We apply our methodology in a case study with an existing conversational recommender system and user simulator, demonstrating that with just a few iterations, we can make the system more robust to conversational breakdowns.
• Score: 15.54070473873364
• License:
• Abstract: We present a methodology to systematically test conversational recommender systems with regards to conversational breakdowns. It involves examining conversations generated between the system and simulated users for a set of pre-defined breakdown types, extracting responsible conversational paths, and characterizing them in terms of the underlying dialogue intents. User simulation offers the advantages of simplicity, cost-effectiveness, and time efficiency for obtaining conversations where potential breakdowns can be identified. The proposed methodology can be used as diagnostic tool as well as a development tool to improve conversational recommendation systems. We apply our methodology in a case study with an existing conversational recommender system and user simulator, demonstrating that with just a few iterations, we can make the system more robust to conversational breakdowns.

Related papers

• Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems [2.788542465279969]
  This paper introduces DAUS, a Domain-Aware User Simulator. We fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment.
  arXiv  Detail & Related papers  (2024-02-20T20:57:47Z)
• Extracting user needs with Chat-GPT for dialogue recommendation [0.0]
  Large-scale language models (LLMs) are becoming increasingly sophisticated and exhibit human-like capabilities. OpenAI's Chat-GPT has a very high inference capability as a dialogue system and the ability to generate high-quality sentences.
  arXiv  Detail & Related papers  (2023-10-30T06:41:26Z)
• In-Context Learning User Simulators for Task-Oriented Dialog Systems [1.7086737326992172]
  This paper presents a novel application of large language models in user simulation for task-oriented dialog systems. By harnessing the power of these models, the proposed approach generates diverse utterances based on user goals and limited dialog examples.
  arXiv  Detail & Related papers  (2023-06-01T15:06:11Z)
• Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models [115.7508325840751]
  The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs) In this paper, we embark on an investigation into the utilization of ChatGPT for conversational recommendation, revealing the inadequacy of the existing evaluation protocol. We propose an interactive Evaluation approach based on LLMs named iEvaLM that harnesses LLM-based user simulators.
  arXiv  Detail & Related papers  (2023-05-22T15:12:43Z)
• GODEL: Large-Scale Pre-Training for Goal-Directed Dialog [119.1397031992088]
  We introduce GODEL, a large pre-trained language model for dialog. We show that GODEL outperforms state-of-the-art pre-trained dialog models in few-shot fine-tuning setups. A novel feature of our evaluation methodology is the introduction of a notion of utility that assesses the usefulness of responses.
  arXiv  Detail & Related papers  (2022-06-22T18:19:32Z)
• Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems [109.16553492049441]
  We propose a novel method to incorporate the knowledge reasoning capability into dialogue systems in a more scalable and generalizable manner. To the best of our knowledge, this is the first work to have transformer models generate responses by reasoning over differentiable knowledge graphs.
  arXiv  Detail & Related papers  (2022-03-20T17:51:49Z)
• Natural Language Understanding for Argumentative Dialogue Systems in the Opinion Building Domain [6.951113351928047]
  This paper introduces a framework for argumentative dialogue systems in the information-seeking domain. Our approach distinguishes multiple user intents and identifies system arguments the user refers to in his or her natural language utterances.
  arXiv  Detail & Related papers  (2021-03-03T21:17:24Z)
• Improving Conversational Question Answering Systems after Deployment using Feedback-Weighted Learning [69.42679922160684]
  We propose feedback-weighted learning based on importance sampling to improve upon an initial supervised system using binary user feedback. Our work opens the prospect to exploit interactions with real users and improve conversational systems after deployment.
  arXiv  Detail & Related papers  (2020-11-01T19:50:34Z)
• Towards Conversational Recommendation over Multi-Type Dialogs [78.52354759386296]
  We propose a new task of conversational recommendation over multi-type dialogs, where the bots can proactively and naturally lead a conversation from a non-recommendation dialog to a recommendation dialog. To facilitate the study of this task, we create a human-to-human Chinese dialog dataset emphDuRecDial (about 10k dialogs, 156k utterances) In each dialog, the recommender proactively leads a multi-type dialog to approach recommendation targets and then makes multiple recommendations with rich interaction behavior.
  arXiv  Detail & Related papers  (2020-05-08T11:01:21Z)
• A Bayesian Approach to Conversational Recommendation Systems [60.12942570608859]
  We present a conversational recommendation system based on a Bayesian approach. A case study based on the application of this approach to emphstagend.com, an online platform for booking entertainers, is discussed.
  arXiv  Detail & Related papers  (2020-02-12T15:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.