A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding
- URL: http://arxiv.org/abs/2304.04256v1
- Date: Sun, 9 Apr 2023 15:28:36 GMT
- Title: A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding
- Authors: Wenbo Pan, Qiguang Chen, Xiao Xu, Wanxiang Che, Libo Qin
- Abstract summary: Zero-shot dialogue understanding aims to enable dialogue to track the user's needs without any training data.
In this work, we investigate the understanding ability of ChatGPT for zero-shot dialogue understanding tasks.
- Score: 55.37338324658501
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Zero-shot dialogue understanding aims to enable dialogue to track the user's
needs without any training data, which has gained increasing attention. In this
work, we investigate the understanding ability of ChatGPT for zero-shot
dialogue understanding tasks including spoken language understanding (SLU) and
dialogue state tracking (DST). Experimental results on four popular benchmarks
reveal the great potential of ChatGPT for zero-shot dialogue understanding. In
addition, extensive analysis shows that ChatGPT benefits from the multi-turn
interactive prompt in the DST task but struggles to perform slot filling for
SLU. Finally, we summarize several unexpected behaviors of ChatGPT in dialogue
understanding tasks, hoping to provide some insights for future research on
building zero-shot dialogue understanding systems with Large Language Models
(LLMs).
Related papers
- OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue
System [43.92593448255296]
We propose an Omnipotent Dialogue pre-training model ( OmniDialog)
It unifies three dialogue tasks into a monolithic framework by multi-task learning, fostering inter-task communication.
We evaluate its performance across four tasks: dialogue summarization, end-to-end dialogue modeling, dialogue state tracking, and intent classification.
arXiv Detail & Related papers (2023-12-28T07:20:49Z) - Self-Explanation Prompting Improves Dialogue Understanding in Large
Language Models [52.24756457516834]
We propose a novel "Self-Explanation" prompting strategy to enhance the comprehension abilities of Large Language Models (LLMs)
This task-agnostic approach requires the model to analyze each dialogue utterance before task execution, thereby improving performance across various dialogue-centric tasks.
Experimental results from six benchmark datasets confirm that our method consistently outperforms other zero-shot prompts and matches or exceeds the efficacy of few-shot prompts.
arXiv Detail & Related papers (2023-09-22T15:41:34Z) - Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue:
An Empirical Study [51.079100495163736]
This paper systematically inspects ChatGPT's performance in two discourse analysis tasks: topic segmentation and discourse parsing.
ChatGPT demonstrates proficiency in identifying topic structures in general-domain conversations yet struggles considerably in specific-domain conversations.
Our deeper investigation indicates that ChatGPT can give more reasonable topic structures than human annotations but only linearly parses the hierarchical rhetorical structures.
arXiv Detail & Related papers (2023-05-15T07:14:41Z) - SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation [55.82577086422923]
We provide a feasible definition of dialogue segmentation points with the help of document-grounded dialogues.
We release a large-scale supervised dataset called SuperDialseg, containing 9,478 dialogues.
We also provide a benchmark including 18 models across five categories for the dialogue segmentation task.
arXiv Detail & Related papers (2023-05-15T06:08:01Z) - ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal,
Causal, and Discourse Relations [52.26802326949116]
We quantitatively evaluate the performance of ChatGPT, an interactive large language model, on inter-sentential relations.
ChatGPT exhibits exceptional proficiency in detecting and reasoning about causal relations.
It is capable of identifying the majority of discourse relations with existing explicit discourse connectives, but the implicit discourse relation remains a formidable challenge.
arXiv Detail & Related papers (2023-04-28T13:14:36Z) - Is ChatGPT Equipped with Emotional Dialogue Capabilities? [14.419588510681773]
The study evaluates the performance of ChatGPT on emotional dialogue understanding and generation through a series of experiments on several downstream tasks.
Our findings indicate that while ChatGPT's performance on emotional dialogue understanding may still lag behind that of supervised models, it exhibits promising results in generating emotional responses.
arXiv Detail & Related papers (2023-04-19T11:42:40Z) - STRUDEL: Structured Dialogue Summarization for Dialogue Comprehension [42.57581945778631]
Abstractive dialogue summarization has long been viewed as an important standalone task in natural language processing.
We propose a novel type of dialogue summarization task - STRUctured DiaLoguE Summarization.
We show that our STRUDEL dialogue comprehension model can significantly improve the dialogue comprehension performance of transformer encoder language models.
arXiv Detail & Related papers (2022-12-24T04:39:54Z) - TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented
Dialogue [113.45485470103762]
In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling.
To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling.
arXiv Detail & Related papers (2020-04-15T04:09:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.