Judgments of research co-created by generative AI: experimental evidence

• URL: http://arxiv.org/abs/2305.11873v1
• Date: Wed, 3 May 2023 15:57:39 GMT
• Title: Judgments of research co-created by generative AI: experimental evidence
• Authors: Pawe{\l} Niszczota, Paul Conway
• Abstract summary: We test whether delegating parts of the research process to LLMs leads people to distrust and devalue researchers and scientific output. People judged delegating to an LLM as less acceptable than delegating to a human. We discuss how this devaluation might transfer into the underreporting of generative AI use.
• Score: 1.8275108630751844
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The introduction of ChatGPT has fuelled a public debate on the use of generative AI (large language models; LLMs), including its use by researchers. In the current work, we test whether delegating parts of the research process to LLMs leads people to distrust and devalue researchers and scientific output. Participants (N=402) considered a researcher who delegates elements of the research process to a PhD student or LLM, and rated (1) moral acceptability, (2) trust in the scientist to oversee future projects, and (3) the accuracy and quality of the output. People judged delegating to an LLM as less acceptable than delegating to a human (d = -0.78). Delegation to an LLM also decreased trust to oversee future research projects (d = -0.80), and people thought the results would be less accurate and of lower quality (d = -0.85). We discuss how this devaluation might transfer into the underreporting of generative AI use.

Related papers

• LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions [20.44227547555244]
  Large language models (LLMs) have led many researchers to consider their usage for scientific work. We present the first large-scale survey of 816 verified research article authors. We find that 81% of researchers have already incorporated LLMs into different aspects of their research workflow.
  arXiv  Detail & Related papers  (2024-10-30T04:25:23Z)
• Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents [64.64280477958283]
  An exponential increase in scientific literature makes it challenging for researchers to stay current with recent advances and identify meaningful research directions. Recent developments in large language models(LLMs) suggest a promising avenue for automating the generation of novel research ideas. We propose a Chain-of-Ideas(CoI) agent, an LLM-based agent that organizes relevant literature in a chain structure to effectively mirror the progressive development in a research domain.
  arXiv  Detail & Related papers  (2024-10-17T03:26:37Z)
• 'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants [13.693069737188859]
  Recent excitement around generative models has sparked a wave of proposals suggesting the replacement of human participation and labor in research and development. We conducted interviews with 19 qualitative researchers to understand their perspectives on this paradigm shift.
  arXiv  Detail & Related papers  (2024-09-28T18:28:47Z)
• Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers [90.26363107905344]
  Large language models (LLMs) have sparked optimism about their potential to accelerate scientific discovery. No evaluations have shown that LLM systems can take the very first step of producing novel, expert-level ideas.
  arXiv  Detail & Related papers  (2024-09-06T08:25:03Z)
• LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing [106.45895712717612]
  Large language models (LLMs) have shown remarkable versatility in various generative tasks. This study focuses on the topic of LLMs assist NLP Researchers. To our knowledge, this is the first work to provide such a comprehensive analysis.
  arXiv  Detail & Related papers  (2024-06-24T01:30:22Z)
• Large language models surpass human experts in predicting neuroscience results [60.26891446026707]
  Large language models (LLMs) forecast novel results better than human experts. BrainBench is a benchmark for predicting neuroscience results. Our approach is not neuroscience-specific and is transferable to other knowledge-intensive endeavors.
  arXiv  Detail & Related papers  (2024-03-04T15:27:59Z)
• Language Models Hallucinate, but May Excel at Fact Verification [89.0833981569957]
  Large language models (LLMs) frequently "hallucinate," resulting in non-factual outputs. Even GPT-3.5 produces factual outputs less than 25% of the time. This underscores the importance of fact verifiers in order to measure and incentivize progress.
  arXiv  Detail & Related papers  (2023-10-23T04:39:01Z)
• Potential Benefits of Employing Large Language Models in Research in Moral Education and Development [0.0]
  Recently, computer scientists have developed large language models (LLMs) by training prediction models with large-scale language corpora and human reinforcements. I will examine how LLMs might contribute to moral education and development research.
  arXiv  Detail & Related papers  (2023-06-23T22:39:05Z)
• Exploring Qualitative Research Using LLMs [8.545798128849091]
  This study aimed to compare and contrast the comprehension capabilities of humans and AI driven large language models. We conducted an experiment with small sample of Alexa app reviews, initially classified by a human analyst. LLMs were then asked to classify these reviews and provide the reasoning behind each classification.
  arXiv  Detail & Related papers  (2023-06-23T05:21:36Z)
• Can Large Language Models Be an Alternative to Human Evaluations? [80.81532239566992]
  Large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation.
  arXiv  Detail & Related papers  (2023-05-03T07:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.