Can GPT-4 learn to analyze moves in research article abstracts?

• URL: http://arxiv.org/abs/2407.15612v2
• Date: Wed, 24 Jul 2024 21:10:24 GMT
• Title: Can GPT-4 learn to analyze moves in research article abstracts?
• Authors: Danni Yu, Marina Bondi, Ken Hyland,
• Abstract summary: We employ the affordances of GPT-4 to automate the annotation process by using natural language prompts. An 8-shot prompt was more effective than one using two, confirming that the inclusion of examples illustrating areas of variability can enhance GPT-4's ability to recognize multiple moves in a single sentence.
• Score: 0.9999629695552195
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: One of the most powerful and enduring ideas in written discourse analysis is that genres can be described in terms of the moves which structure a writer's purpose. Considerable research has sought to identify these distinct communicative acts, but analyses have been beset by problems of subjectivity, reliability and the time-consuming need for multiple coders to confirm analyses. In this paper we employ the affordances of GPT-4 to automate the annotation process by using natural language prompts. Focusing on abstracts from articles in four applied linguistics journals, we devise prompts which enable the model to identify moves effectively. The annotated outputs of these prompts were evaluated by two assessors with a third addressing disagreements. The results show that an 8-shot prompt was more effective than one using two, confirming that the inclusion of examples illustrating areas of variability can enhance GPT-4's ability to recognize multiple moves in a single sentence and reduce bias related to textual position. We suggest that GPT-4 offers considerable potential in automating this annotation process, when human actors with domain specific linguistic expertise inform the prompting process.

Related papers

• GPT Assisted Annotation of Rhetorical and Linguistic Features for Interpretable Propaganda Technique Detection in News Text [1.2699007098398802]
  This study codifies 22 rhetorical and linguistic features identified in literature related to the language of persuasion. RhetAnn, a web application, was specifically designed to minimize an otherwise considerable mental effort. A small set of annotated data was used to fine-tune GPT-3.5, a generative large language model (LLM), to annotate the remaining data.
  arXiv  Detail & Related papers  (2024-07-16T15:15:39Z)
• Large Language Models for Propaganda Span Annotation [11.64165958410489]
  We investigate whether large language models (LLMs), such as GPT-4, can effectively perform the task. We use a large-scale in-house dataset consisting of annotations from human annotators with varying expertise levels. We plan to make the collected span-level labels from multiple annotators, including GPT-4, available for the community.
  arXiv  Detail & Related papers  (2023-11-16T11:37:54Z)
• GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks [70.98062518872999]
  We validate GPT-4V's capabilities for evaluation purposes, addressing tasks ranging from foundational image-to-text and text-to-image synthesis to high-level image-to-image translations and multi-images to text alignment. Notably, GPT-4V shows promising agreement with humans across various tasks and evaluation methods, demonstrating immense potential for multi-modal LLMs as evaluators.
  arXiv  Detail & Related papers  (2023-11-02T16:11:09Z)
• "You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation [60.863629647985526]
  We examine the successes and limitations of the GPT-3, ChatGPT, and GPT-4 models in analysis of sentence meaning structure. We find that models can reliably reproduce the basic format of AMR, and can often capture core event, argument, and modifier structure. Overall, our findings indicate that these models out-of-the-box can capture aspects of semantic structure, but there remain key limitations in their ability to support fully accurate semantic analyses or parses.
  arXiv  Detail & Related papers  (2023-10-26T21:47:59Z)
• Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise? [0.8924669503280334]
  GPT-4, prompted with annotation guidelines, performs on par with well-trained law student annotators. We demonstrated how to analyze GPT-4's predictions to identify and mitigate deficiencies in annotation guidelines.
  arXiv  Detail & Related papers  (2023-06-24T08:48:24Z)
• PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts [76.18347405302728]
  This study uses a plethora of adversarial textual attacks targeting prompts across multiple levels: character, word, sentence, and semantic. The adversarial prompts are then employed in diverse tasks including sentiment analysis, natural language inference, reading comprehension, machine translation, and math problem-solving. Our findings demonstrate that contemporary Large Language Models are not robust to adversarial prompts.
  arXiv  Detail & Related papers  (2023-06-07T15:37:00Z)
• Document-Level Machine Translation with Large Language Models [91.03359121149595]
  Large language models (LLMs) can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. This paper provides an in-depth evaluation of LLMs' ability on discourse modeling.
  arXiv  Detail & Related papers  (2023-04-05T03:49:06Z)
• Prompting Large Language Model for Machine Translation: A Case Study [87.88120385000666]
  We offer a systematic study on prompting strategies for machine translation. We examine factors for prompt template and demonstration example selection. We explore the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning.
  arXiv  Detail & Related papers  (2023-01-17T18:32:06Z)
• Instruction Tuning for Few-Shot Aspect-Based Sentiment Analysis [72.9124467710526]
  generative approaches have been proposed to extract all four elements as (one or more) quadruplets from text as a single task. We propose a unified framework for solving ABSA, and the associated sub-tasks to improve the performance in few-shot scenarios.
  arXiv  Detail & Related papers  (2022-10-12T23:38:57Z)
• Polling Latent Opinions: A Method for Computational Sociolinguistics Using Transformer Language Models [4.874780144224057]
  We use the capacity for memorization and extrapolation of Transformer Language Models to learn the linguistic behaviors of a subgroup within larger corpora of Yelp reviews. We show that even in cases where a specific keyphrase is limited or not present at all in the training corpora, the GPT is able to accurately generate large volumes of text that have the correct sentiment.
  arXiv  Detail & Related papers  (2022-04-15T14:33:58Z)
• Automatic Analysis of Linguistic Features in Journal Articles of Different Academic Impacts with Feature Engineering Techniques [0.975434908987426]
  This study attempts to extract micro-level linguistic features in high- and moderate-impact journal RAs, using feature engineering methods. We extracted 25 highly relevant features from the Corpus of English Journal Articles through feature selection methods. Results showed that 24 linguistic features such as the overlapping of content words between adjacent sentences, the use of third-person pronouns, auxiliary verbs, tense, emotional words provide consistent and accurate predictions for journal articles with different academic impacts.
  arXiv  Detail & Related papers  (2021-11-15T03:56:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.