The Battle of LLMs: A Comparative Study in Conversational QA Tasks

• URL: http://arxiv.org/abs/2405.18344v1
• Date: Tue, 28 May 2024 16:42:43 GMT
• Title: The Battle of LLMs: A Comparative Study in Conversational QA Tasks
• Authors: Aryan Rangapur, Aman Rangapur,
• Abstract summary: This research delves into the responses generated by ChatGPT, GPT-4, Gemini, Mixtral and Claude across different Conversational QA corpora. Evaluation scores were meticulously computed and subsequently compared to ascertain the overall performance of these models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models have gained considerable interest for their impressive performance on various tasks. Within this domain, ChatGPT and GPT-4, developed by OpenAI, and the Gemini, developed by Google, have emerged as particularly popular among early adopters. Additionally, Mixtral by Mistral AI and Claude by Anthropic are newly released, further expanding the landscape of advanced language models. These models are viewed as disruptive technologies with applications spanning customer service, education, healthcare, and finance. More recently, Mistral has entered the scene, captivating users with its unique ability to generate creative content. Understanding the perspectives of these users is crucial, as they can offer valuable insights into the potential strengths, weaknesses, and overall success or failure of these technologies in various domains. This research delves into the responses generated by ChatGPT, GPT-4, Gemini, Mixtral and Claude across different Conversational QA corpora. Evaluation scores were meticulously computed and subsequently compared to ascertain the overall performance of these models. Our study pinpointed instances where these models provided inaccurate answers to questions, offering insights into potential areas where they might be susceptible to errors. In essence, this research provides a comprehensive comparison and evaluation of these state of-the-art language models, shedding light on their capabilities while also highlighting potential areas for improvement

Related papers

• Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases [98.35348038111508]
  This paper presents an in-depth comparative study of two pioneering models: Google's Gemini and OpenAI's GPT-4V(ision) The core of our analysis delves into the distinct visual comprehension abilities of each model. Our findings illuminate the unique strengths and niches of both models.
  arXiv  Detail & Related papers  (2023-12-22T18:59:58Z)
• Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principles [2.134057414078079]
  Large Language Models (LLMs) gain widespread use, ensuring their security and robustness is critical. This paper presents a novel study focusing on exploitation of such large language models against deceptive interactions. Our results demonstrate a significant finding in that these large language models are susceptible to deception and social engineering attacks.
  arXiv  Detail & Related papers  (2023-11-24T23:57:44Z)
• MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration [102.41118020705876]
  Large Language Models (LLMs) have marked a significant advancement in the field of natural language processing. As their applications extend into multi-agent environments, a need has arisen for a comprehensive evaluation framework. This work introduces a novel benchmarking framework specifically tailored to assess LLMs within multi-agent settings.
  arXiv  Detail & Related papers  (2023-11-14T21:46:27Z)
• MMBench: Is Your Multi-modal Model an All-around Player? [114.45702807380415]
  We propose MMBench, a benchmark for assessing the multi-modal capabilities of vision-language models. MMBench is meticulously curated with well-designed quality control schemes. MMBench incorporates multiple-choice questions in both English and Chinese versions.
  arXiv  Detail & Related papers  (2023-07-12T16:23:09Z)
• Improving Factuality and Reasoning in Language Models through Multiagent Debate [95.10641301155232]
  We present a complementary approach to improve language responses where multiple language model instances propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer. Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks. Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate.
  arXiv  Detail & Related papers  (2023-05-23T17:55:11Z)
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations [91.98516412612739]
  We first provide a systematically designed, diverse, informative, large-scale dataset of instructional conversations, UltraChat. Our objective is to capture the breadth of interactions that a human might have with an AI assistant. We fine-tune a LLaMA model to create a powerful conversational model, UltraLLaMA.
  arXiv  Detail & Related papers  (2023-05-23T16:49:14Z)
• A Glimpse in ChatGPT Capabilities and its impact for AI research [4.2245880148320705]
  Large language models (LLMs) have recently become a popular topic in the field of Artificial Intelligence (AI) research. These models are trained on massive amounts of data and can be used for a wide range of tasks, including language translation, text generation, and question answering.
  arXiv  Detail & Related papers  (2023-05-10T12:10:51Z)
• ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking about [15.19126287569545]
  This research examines the responses generated by ChatGPT from different Conversational QA corpora. The study employed BERT similarity scores to compare these responses with correct answers and obtain Natural Language Inference(NLI) labels. The study identified instances where ChatGPT provided incorrect answers to questions, providing insights into areas where the model may be prone to error.
  arXiv  Detail & Related papers  (2023-04-06T18:42:47Z)
• Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defense [3.8702319399328466]
  Review aims to provide a brief overview of the history, state of the art, and implications of Generative Language Models in terms of their principles, abilities, limitations, and future prospects. Generative Language Models gained significant attention in late 2022 / early 2023, notably with the introduction of models refined to act consistently with users' expectations of interactions with AI (conversational models) This review aims to provide a brief overview of the history, state of the art, and implications of Generative Language Models in terms of their principles, abilities, limitations, and future prospects.
  arXiv  Detail & Related papers  (2023-03-21T18:45:09Z)
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models [648.3665819567409]
  Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Big-bench consists of 204 tasks, contributed by 450 authors across 132 institutions. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench.
  arXiv  Detail & Related papers  (2022-06-09T17:05:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.