A Transformer-Based User Satisfaction Prediction for Proactive Interaction Mechanism in DuerOS

• URL: http://arxiv.org/abs/2212.03817v1
• Date: Mon, 5 Dec 2022 09:17:49 GMT
• Title: A Transformer-Based User Satisfaction Prediction for Proactive Interaction Mechanism in DuerOS
• Authors: Wei Shen, Xiaonan He, Chuheng Zhang, Xuyun Zhang, Jian XIe
• Abstract summary: We propose a proactive interaction mechanism where the system predicts the user satisfaction with the candidate response before giving it to the user. If the user is not likely to be satisfied according to the prediction, the system will ask the user a suitable question to determine the real intent of the user. We deploy and evaluate our model on DuerOS, and observe a 19% relative improvement on the accuracy of user satisfaction prediction and 2.3% relative improvement on user experience.
• Score: 12.060990859604681
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, spoken dialogue systems have been widely deployed in a variety of applications, serving a huge number of end-users. A common issue is that the errors resulting from noisy utterances, semantic misunderstandings, or lack of knowledge make it hard for a real system to respond properly, possibly leading to an unsatisfactory user experience. To avoid such a case, we consider a proactive interaction mechanism where the system predicts the user satisfaction with the candidate response before giving it to the user. If the user is not likely to be satisfied according to the prediction, the system will ask the user a suitable question to determine the real intent of the user instead of providing the response directly. With such an interaction with the user, the system can give a better response to the user. Previous models that predict the user satisfaction are not applicable to DuerOS which is a large-scale commercial dialogue system. They are based on hand-crafted features and thus can hardly learn the complex patterns lying behind millions of conversations and temporal dependency in multiple turns of the conversation. Moreover, they are trained and evaluated on the benchmark datasets with adequate labels, which are expensive to obtain in a commercial dialogue system. To face these challenges, we propose a pipeline to predict the user satisfaction to help DuerOS decide whether to ask for clarification in each turn. Specifically, we propose to first generate a large number of weak labels and then train a transformer-based model to predict the user satisfaction with these weak labels. Empirically, we deploy and evaluate our model on DuerOS, and observe a 19% relative improvement on the accuracy of user satisfaction prediction and 2.3% relative improvement on user experience.

Related papers

• Towards Empathetic Conversational Recommender Systems [77.53167131692]
  We propose an empathetic conversational recommender (ECR) framework. ECR contains two main modules: emotion-aware item recommendation and emotion-aligned response generation. Our experiments on the ReDial dataset validate the efficacy of our framework in enhancing recommendation accuracy and improving user satisfaction.
  arXiv  Detail & Related papers  (2024-08-30T15:43:07Z)
• System-2 Recommenders: Disentangling Utility and Engagement in Recommendation Systems via Temporal Point-Processes [80.97898201876592]
  We propose a generative model in which past content interactions impact the arrival rates of users based on a self-exciting Hawkes process. We show analytically that given samples it is possible to disentangle System-1 and System-2 and allow content optimization based on user utility.
  arXiv  Detail & Related papers  (2024-05-29T18:19:37Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems [60.27663010453209]
  We leverage large language models (LLMs) to generate satisfaction-aware counterfactual dialogues. We gather human annotations to ensure the reliability of the generated samples. Our results shed light on the need for data augmentation approaches for user satisfaction estimation in TOD systems.
  arXiv  Detail & Related papers  (2024-03-27T23:45:31Z)
• Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluation [17.41434948048325]
  We conduct an interactive user study to unveil how vulnerable TOD systems are against realistic scenarios. Our study reveals that conversations in open-goal settings lead to catastrophic failures of the system. We discover a novel "pretending" behavior, in which the system pretends to handle the user requests even though they are beyond the system's capabilities.
  arXiv  Detail & Related papers  (2023-05-23T09:24:53Z)
• Latent User Intent Modeling for Sequential Recommenders [92.66888409973495]
  Sequential recommender models learn to predict the next items a user is likely to interact with based on his/her interaction history on the platform. Most sequential recommenders however lack a higher-level understanding of user intents, which often drive user behaviors online. Intent modeling is thus critical for understanding users and optimizing long-term user experience.
  arXiv  Detail & Related papers  (2022-11-17T19:00:24Z)
• Multi-stage Clarification in Conversational AI: The case of Question-Answering Dialogue Systems [0.27998963147546135]
  Clarification resolution plays an important role in various information retrieval tasks such as interactive question answering and conversational search. We propose a multi-stage clarification mechanism for prompting clarification and query selection in the context of a question answering dialogue system. Our proposed mechanism improves the overall user experience and outperforms competitive baselines with two datasets.
  arXiv  Detail & Related papers  (2021-10-28T15:45:44Z)
• Improving Conversational Question Answering Systems after Deployment using Feedback-Weighted Learning [69.42679922160684]
  We propose feedback-weighted learning based on importance sampling to improve upon an initial supervised system using binary user feedback. Our work opens the prospect to exploit interactions with real users and improve conversational systems after deployment.
  arXiv  Detail & Related papers  (2020-11-01T19:50:34Z)
• User Intention Recognition and Requirement Elicitation Method for Conversational AI Services [7.941589241861105]
  We aim to obtain user requirements as accurately as possible in as few rounds as possible. A user intention recognition method based on Knowledge Graph (KG) was developed for fuzzy requirement inference. A requirement elicitation method based on Granular Computing was proposed for dialog policy generation.
  arXiv  Detail & Related papers  (2020-09-03T08:26:39Z)
• Predict-then-Decide: A Predictive Approach for Wait or Answer Task in Dialogue Systems [24.560203199376478]
  We propose a predictive approach named Predict-then-Decide (PTD) to tackle this Wait-or-Answer problem. We conduct experiments on two real-life scenarios and three public datasets.
  arXiv  Detail & Related papers  (2020-05-27T01:48:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.