Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
- URL: http://arxiv.org/abs/2305.16798v1
- Date: Fri, 26 May 2023 10:19:30 GMT
- Title: Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
- Authors: Yue Feng, Yunlong Jiao, Animesh Prasad, Nikolaos Aletras, Emine
Yilmaz, Gabriella Kazai
- Abstract summary: We propose SG-USM, a novel schema-guided user satisfaction modeling framework.
It explicitly models the degree to which the user's preferences regarding the task attributes are fulfilled by the system for predicting the user's satisfaction level.
- Score: 38.251046341024455
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: User Satisfaction Modeling (USM) is one of the popular choices for
task-oriented dialogue systems evaluation, where user satisfaction typically
depends on whether the user's task goals were fulfilled by the system.
Task-oriented dialogue systems use task schema, which is a set of task
attributes, to encode the user's task goals. Existing studies on USM neglect
explicitly modeling the user's task goals fulfillment using the task schema. In
this paper, we propose SG-USM, a novel schema-guided user satisfaction modeling
framework. It explicitly models the degree to which the user's preferences
regarding the task attributes are fulfilled by the system for predicting the
user's satisfaction level. SG-USM employs a pre-trained language model for
encoding dialogue context and task attributes. Further, it employs a
fulfillment representation layer for learning how many task attributes have
been fulfilled in the dialogue, an importance predictor component for
calculating the importance of task attributes. Finally, it predicts the user
satisfaction based on task attribute fulfillment and task attribute importance.
Experimental results on benchmark datasets (i.e. MWOZ, SGD, ReDial, and JDDC)
show that SG-USM consistently outperforms competitive existing methods. Our
extensive analysis demonstrates that SG-USM can improve the interpretability of
user satisfaction modeling, has good scalability as it can effectively deal
with unseen tasks and can also effectively work in low-resource settings by
leveraging unlabeled data.
Related papers
- Many Hands Make Light Work: Task-Oriented Dialogue System with Module-Based Mixture-of-Experts [9.129081545049992]
Task-oriented dialogue systems have greatly benefited from pre-trained language models (PLMs)
We propose Soft Mixture-of-Expert Task-Oriented Dialogue system (SMETOD)
SMETOD leverages an ensemble of Mixture-of-Experts (MoEs) to excel at subproblems and generate specialized outputs for task-oriented dialogues.
We extensively evaluate our model on three benchmark functionalities: intent prediction, dialogue state tracking, and dialogue response generation.
arXiv Detail & Related papers (2024-05-16T01:02:09Z) - CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems [60.27663010453209]
We leverage large language models (LLMs) to generate satisfaction-aware counterfactual dialogues.
We gather human annotations to ensure the reliability of the generated samples.
Our results shed light on the need for data augmentation approaches for user satisfaction estimation in TOD systems.
arXiv Detail & Related papers (2024-03-27T23:45:31Z) - Tell Me More! Towards Implicit User Intention Understanding of Language
Model Driven Agents [110.25679611755962]
Current language model-driven agents often lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions.
We introduce Intention-in-Interaction (IN3), a novel benchmark designed to inspect users' implicit intentions through explicit queries.
We empirically train Mistral-Interact, a powerful model that proactively assesses task vagueness, inquires user intentions, and refines them into actionable goals.
arXiv Detail & Related papers (2024-02-14T14:36:30Z) - TaskBench: Benchmarking Large Language Models for Task Automation [85.3879908356586]
We introduce TaskBench to evaluate the capability of large language models in task automation.
To generate high-quality evaluation datasets, we introduce the concept of Tool Graph.
We also propose TaskEval to evaluate the capability of LLMs from different aspects, including task decomposition, tool invocation, and parameter prediction.
arXiv Detail & Related papers (2023-11-30T18:02:44Z) - Task Relation-aware Continual User Representation Learning [26.514449669395297]
Previous efforts in user modeling mainly focus on learning a task-specific user representation that is designed for a single task.
Recent studies introduce the concept of universal user representation, which is a more generalized representation of a user relevant to a variety of tasks.
Despite their effectiveness, existing approaches for learning universal user representations are impractical in real-world applications.
We propose a novel continual user representation learning method, called TERACON, whose learning capability is not limited as the number of learned tasks increases.
arXiv Detail & Related papers (2023-06-01T08:10:03Z) - Meta-Reinforcement Learning Based on Self-Supervised Task Representation
Learning [23.45043290237396]
MoSS is a context-based Meta-reinforcement learning algorithm based on Self-Supervised task representation learning.
On MuJoCo and Meta-World benchmarks, MoSS outperforms prior in terms of performance, sample efficiency (3-50x faster), adaptation efficiency, and generalization.
arXiv Detail & Related papers (2023-04-29T15:46:19Z) - Task Aware Dreamer for Task Generalization in Reinforcement Learning [32.93706056123124]
We show that training a general world model can utilize similar structures in tasks and help train more generalizable agents.
We introduce a novel method named Task Aware Dreamer (TAD), which integrates reward-informed features to identify latent characteristics across tasks.
Experiments in both image-based and state-based tasks show that TAD can significantly improve the performance of handling different tasks simultaneously.
arXiv Detail & Related papers (2023-03-09T08:04:16Z) - Task Adaptive Parameter Sharing for Multi-Task Learning [114.80350786535952]
Adaptive Task Adapting Sharing (TAPS) is a method for tuning a base model to a new task by adaptively modifying a small, task-specific subset of layers.
Compared to other methods, TAPS retains high accuracy on downstream tasks while introducing few task-specific parameters.
We evaluate our method on a suite of fine-tuning tasks and architectures (ResNet, DenseNet, ViT) and show that it achieves state-of-the-art performance while being simple to implement.
arXiv Detail & Related papers (2022-03-30T23:16:07Z) - Exploring Relational Context for Multi-Task Dense Prediction [76.86090370115]
We consider a multi-task environment for dense prediction tasks, represented by a common backbone and independent task-specific heads.
We explore various attention-based contexts, such as global and local, in the multi-task setting.
We propose an Adaptive Task-Relational Context module, which samples the pool of all available contexts for each task pair.
arXiv Detail & Related papers (2021-04-28T16:45:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.