ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?

• URL: http://arxiv.org/abs/2309.08636v3
• Date: Mon, 16 Oct 2023 14:24:02 GMT
• Title: ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?
• Abstract summary: ChatGPT-4 showed the highest quantitative accuracy, closely followed by ChatGPT-3.5, Bing, and Bard. All AIs exhibited proficiency in merging existing knowledge, but none produced original scientific content.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Historical emphasis on writing mastery has shifted with advances in generative AI, especially in scientific writing. This study analysed six AI chatbots for scholarly writing in humanities and archaeology. Using methods that assessed factual correctness and scientific contribution, ChatGPT-4 showed the highest quantitative accuracy, closely followed by ChatGPT-3.5, Bing, and Bard. However, Claude 2 and Aria scored considerably lower. Qualitatively, all AIs exhibited proficiency in merging existing knowledge, but none produced original scientific content. Inter-estingly, our findings suggest ChatGPT-4 might represent a plateau in large language model size. This research emphasizes the unique, intricate nature of human research, suggesting that AI's emulation of human originality in scientific writing is challenging. As of 2023, while AI has transformed content generation, it struggles with original contributions in humanities. This may change as AI chatbots continue to evolve into LLM-powered software.

Related papers

• The AI-Enabled Scientific Frontier [1.6725669564247125]
  We assemble a corpus of 2,507 head-to-head comparisons between AI and other scientific analysis techniques.<n>Relative to traditional statistics, AI often outperforms, but at a significantly higher computational cost.<n>But there are also nearly a quarter of cases where AI is both more expensive and performs worse than traditional statistical techniques.
  arXiv  Detail & Related papers  (2026-09-14T19:22:54Z)
• On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists [113.03797263688519]
  Many scientists view AI reviewers as probabilistic systems without the expertise to evaluate research.<n>Existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts.
  arXiv  Detail & Related papers  (2026-05-20T03:33:55Z)
• Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing [55.2480439325792]
  Misclassification can lead to false plagiarism accusations and misleading claims about AI prevalence in online content. We systematically evaluate eleven state-of-the-art AI-text detectors using our AI-Polished-Text Evaluation dataset. Our findings reveal that detectors frequently misclassify even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models.
  arXiv  Detail & Related papers  (2025-02-21T18:45:37Z)
• Evaluating Sakana's AI Scientist for Autonomous Research: Wishful Thinking or an Emerging Reality Towards 'Artificial Research Intelligence' (ARI)? [19.524056927240498]
  Sakana recently introduced the 'AI Scientist', claiming to conduct research autonomously, i.e. they imply to have achieved what we term Artificial Research Intelligence (ARI) Our evaluation of the AI Scientist reveals critical shortcomings.
  arXiv  Detail & Related papers  (2025-02-20T06:22:03Z)
• AI Expands Scientists' Impact but Contracts Science's Focus [15.771625061196927]
  We show accelerated adoption of AI among scientists and consistent professional advantages associated with AI use.<n>Scientists who engage in AI-augmented research publish 3.02 times more papers, receive 4.84 times more citations.<n>By contrast, AI research shrinks the collective volume of scientific topics studied by 4.63%.
  arXiv  Detail & Related papers  (2024-12-10T18:24:17Z)
• Human Bias in the Face of AI: The Role of Human Judgement in AI Generated Text Evaluation [48.70176791365903]
  This study explores how bias shapes the perception of AI versus human generated content. We investigated how human raters respond to labeled and unlabeled content.
  arXiv  Detail & Related papers  (2024-09-29T04:31:45Z)
• Empirical evidence of Large Language Model's influence on human spoken communication [25.09136621615789]
  Artificial Intelligence (AI) agents now interact with billions of humans in natural language. This raises the question of whether AI has the potential to shape a fundamental aspect of human culture: the way we speak. Recent analyses revealed that scientific publications already exhibit evidence of AI-specific language.
  arXiv  Detail & Related papers  (2024-09-03T10:01:51Z)
• ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing [45.187790784934734]
  This paper focuses on Conversational XAI for AI-assisted scientific writing tasks. We identify four design rationales: "multifaceted", "controllability", "mix-initiative", "context-aware drill-down" We incorporate them into an interactive prototype, ConvXAI, which facilitates heterogeneous AI explanations for scientific writing through dialogue.
  arXiv  Detail & Related papers  (2023-05-16T19:48:49Z)
• AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays [66.36541161082856]
  ChatGPT and similar generative AI models have attracted hundreds of millions of users. This study compares human-written versus ChatGPT-generated argumentative student essays.
  arXiv  Detail & Related papers  (2023-04-24T12:58:28Z)
• One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era [95.2284704286191]
  GPT-4 (a.k.a. ChatGPT plus) is one small step for generative AI (GAI) but one giant leap for artificial general intelligence (AGI) Since its official release in November 2022, ChatGPT has quickly attracted numerous users with extensive media coverage. This work is the first to survey ChatGPT with a comprehensive review of its underlying technology, applications, and challenges.
  arXiv  Detail & Related papers  (2023-04-04T06:22:09Z)
• Towards Healthy AI: Large Language Models Need Therapists Too [41.86344997530743]
  We define Healthy AI to be safe, trustworthy and ethical. We present the SafeguardGPT framework that uses psychotherapy to correct for these harmful behaviors.
  arXiv  Detail & Related papers  (2023-04-02T00:39:12Z)
• ChatGPT or academic scientist? Distinguishing authorship with over 99% accuracy using off-the-shelf machine learning tools [0.0]
  ChatGPT has enabled access to AI-generated writing for the masses. Need to discriminate human writing from AI is now both critical and urgent. We developed a method for discriminating text generated by ChatGPT from (human) academic scientists.
  arXiv  Detail & Related papers  (2023-03-28T23:16:00Z)
• A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need? [112.12974778019304]
  generative AI (AIGC, a.k.a AI-generated content) has made headlines everywhere because of its ability to analyze and create text, images, and beyond. In the era of AI transitioning from pure analysis to creation, it is worth noting that ChatGPT, with its most recent language model GPT-4, is just a tool out of numerous AIGC tasks. This work focuses on the technological development of various AIGC tasks based on their output type, including text, images, videos, 3D content, etc.
  arXiv  Detail & Related papers  (2023-03-21T10:09:47Z)
• Can Machines Imitate Humans? Integrative Turing-like tests for Language and Vision Demonstrate a Narrowing Gap [56.611702960809644]
  We benchmark AI's ability to imitate humans in three language tasks and three vision tasks.<n>Next, we conducted 72,191 Turing-like tests with 1,916 human judges and 10 AI judges.<n>Imitation ability showed minimal correlation with conventional AI performance metrics.
  arXiv  Detail & Related papers  (2022-11-23T16:16:52Z)
• Trustworthy AI: A Computational Perspective [54.80482955088197]
  We focus on six of the most crucial dimensions in achieving trustworthy AI: (i) Safety & Robustness, (ii) Non-discrimination & Fairness, (iii) Explainability, (iv) Privacy, (v) Accountability & Auditability, and (vi) Environmental Well-Being. For each dimension, we review the recent related technologies according to a taxonomy and summarize their applications in real-world systems.
  arXiv  Detail & Related papers  (2021-07-12T14:21:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.