Is ChatGPT a Good Multi-Party Conversation Solver?

• URL: http://arxiv.org/abs/2310.16301v1
• Date: Wed, 25 Oct 2023 02:18:40 GMT
• Title: Is ChatGPT a Good Multi-Party Conversation Solver?
• Abstract summary: Large Language Models (LLMs) have emerged as influential instruments within the realm of natural language processing. In this paper, we delve into the potential of generative LLMs within the context of multi-party conversations. An empirical analysis is conducted to assess the zero-shot learning capabilities of ChatGPT and GPT-4.
• Score: 42.939861319863795
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have emerged as influential instruments within the realm of natural language processing; nevertheless, their capacity to handle multi-party conversations (MPCs) -- a scenario marked by the presence of multiple interlocutors involved in intricate information exchanges -- remains uncharted. In this paper, we delve into the potential of generative LLMs such as ChatGPT and GPT-4 within the context of MPCs. An empirical analysis is conducted to assess the zero-shot learning capabilities of ChatGPT and GPT-4 by subjecting them to evaluation across three MPC datasets that encompass five representative tasks. The findings reveal that ChatGPT's performance on a number of evaluated MPC tasks leaves much to be desired, whilst GPT-4's results portend a promising future. Additionally, we endeavor to bolster performance through the incorporation of MPC structures, encompassing both speaker and addressee architecture. This study provides an exhaustive evaluation and analysis of applying generative LLMs to MPCs, casting a light upon the conception and creation of increasingly effective and robust MPC agents. Concurrently, this work underscores the challenges implicit in the utilization of LLMs for MPCs, such as deciphering graphical information flows and generating stylistically consistent responses.

Related papers

• Let's Use ChatGPT To Write Our Paper! Benchmarking LLMs To Write the Introduction of a Research Paper [64.50822834679101]
  SciIG is a task that evaluates LLMs' ability to produce coherent introductions from titles, abstracts, and related works.<n>We assess five state-of-the-art models, including open-source (DeepSeek-v3, Gemma-3-12B, LLaMA 4-Maverick, MistralAI Small 3.1) and closed-source GPT-4o systems.<n>Results demonstrate LLaMA-4 Maverick's superior performance on most metrics, particularly in semantic similarity and faithfulness.
  arXiv  Detail & Related papers  (2025-08-19T21:11:11Z)
• Don't Stop the Multi-Party! On Generating Synthetic Multi-Party Conversations with Constraints [11.566214724241798]
  Multi-Party Conversations (MPCs) are widely studied across disciplines, with social media as a primary data source due to their accessibility. This work explores the feasibility of generating diverse MPCs with instruction-tuned Large Language Models.
  arXiv  Detail & Related papers  (2025-02-19T10:10:43Z)
• Exploring the Potential of Large Language Models in Computational Argumentation [54.85665903448207]
  Large language models (LLMs) have demonstrated impressive capabilities in understanding context and generating natural language. This work aims to embark on an assessment of LLMs, such as ChatGPT, Flan models, and LLaMA2 models, in both zero-shot and few-shot settings.
  arXiv  Detail & Related papers  (2023-11-15T15:12:15Z)
• GLoRE: Evaluating Logical Reasoning of Large Language Models [29.914546407784552]
  We introduce GLoRE, a benchmark comprised of 12 datasets that span three different types of tasks. ChatGPT and GPT-4 show a strong capability of logical reasoning, with GPT-4 surpassing ChatGPT by a large margin. We propose a self-consistency probing method to enhance the accuracy of ChatGPT and a fine-tuned method to boost the performance of an open LLM.
  arXiv  Detail & Related papers  (2023-10-13T13:52:15Z)
• Metacognitive Prompting Improves Understanding in Large Language Models [12.112914393948415]
  We introduce Metacognitive Prompting (MP), a strategy inspired by human introspective reasoning processes. We conduct experiments on four prevalent Large Language Models (LLMs) across ten natural language understanding (NLU) datasets. MP consistently outperforms existing prompting methods in both general and domain-specific NLU tasks.
  arXiv  Detail & Related papers  (2023-08-10T05:10:17Z)
• Pushing the Limits of ChatGPT on NLP Tasks [79.17291002710517]
  Despite the success of ChatGPT, its performances on most NLP tasks are still well below the supervised baselines. In this work, we looked into the causes, and discovered that its subpar performance was caused by the following factors. We propose a collection of general modules to address these issues, in an attempt to push the limits of ChatGPT on NLP tasks.
  arXiv  Detail & Related papers  (2023-06-16T09:40:05Z)
• Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT and GPT-4 for Mining Insights at Scale [4.6542291555324296]
  We evaluate the combination of ChatGPT and GPT-4 for the real-world task of mining insights from a text corpus. We extract 4,392 research challenges in over 100 topics from the 2023CHI conference proceedings. We conclude that the combination of ChatGPT and GPT-4 makes an excellent cost-efficient means for analyzing a text corpus at scale.
  arXiv  Detail & Related papers  (2023-06-08T08:41:30Z)
• A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets [19.521390684403293]
  We present a thorough evaluation of ChatGPT's performance on diverse academic datasets. Specifically, we evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in these datasets.
  arXiv  Detail & Related papers  (2023-05-29T12:37:21Z)
• Document-Level Machine Translation with Large Language Models [91.03359121149595]
  Large language models (LLMs) can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. This paper provides an in-depth evaluation of LLMs' ability on discourse modeling.
  arXiv  Detail & Related papers  (2023-04-05T03:49:06Z)
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity [79.12003701981092]
  We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-02-08T12:35:34Z)
• MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding [58.95156916558384]
  We present MPC-BERT, a pre-trained model for MPC understanding. We evaluate MPC-BERT on three downstream tasks including addressee recognition, speaker identification and response selection.
  arXiv  Detail & Related papers  (2021-06-03T01:49:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.