Large Language Models for Propaganda Detection
- URL: http://arxiv.org/abs/2310.06422v2
- Date: Mon, 27 Nov 2023 10:18:36 GMT
- Title: Large Language Models for Propaganda Detection
- Authors: Kilian Sprenkamp, Daniel Gordon Jones, Liudmila Zavolokina
- Abstract summary: This study investigates the effectiveness of Large Language Models (LLMs) for propaganda detection.
Five variations of GPT-3 and GPT-4 are employed, incorporating various prompt engineering and fine-tuning strategies.
Our findings demonstrate that GPT-4 achieves comparable results to the current state-of-the-art.
- Score: 2.587450057509126
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The prevalence of propaganda in our digital society poses a challenge to
societal harmony and the dissemination of truth. Detecting propaganda through
NLP in text is challenging due to subtle manipulation techniques and contextual
dependencies. To address this issue, we investigate the effectiveness of modern
Large Language Models (LLMs) such as GPT-3 and GPT-4 for propaganda detection.
We conduct experiments using the SemEval-2020 task 11 dataset, which features
news articles labeled with 14 propaganda techniques as a multi-label
classification problem. Five variations of GPT-3 and GPT-4 are employed,
incorporating various prompt engineering and fine-tuning strategies across the
different models. We evaluate the models' performance by assessing metrics such
as $F1$ score, $Precision$, and $Recall$, comparing the results with the
current state-of-the-art approach using RoBERTa. Our findings demonstrate that
GPT-4 achieves comparable results to the current state-of-the-art. Further,
this study analyzes the potential and challenges of LLMs in complex tasks like
propaganda detection.
Related papers
- Can GPT-4 learn to analyze moves in research article abstracts? [0.9999629695552195]
We employ the affordances of GPT-4 to automate the annotation process by using natural language prompts.
An 8-shot prompt was more effective than one using two, confirming that the inclusion of examples illustrating areas of variability can enhance GPT-4's ability to recognize multiple moves in a single sentence.
arXiv Detail & Related papers (2024-07-22T13:14:27Z) - Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda
Spans in News Articles [11.64165958410489]
We develop the largest propaganda dataset to date, comprised of 8K paragraphs from newspaper articles, labeled at the text span level following a taxonomy of 23 propagandistic techniques.
Our work offers the first attempt to understand the performance of large language models (LLMs), using GPT-4, for fine-grained propaganda detection from text.
Results showed that GPT-4's performance degrades as the task moves from simply classifying a paragraph as propagandistic or not, to the fine-grained task of detecting propaganda techniques and their manifestation in text.
arXiv Detail & Related papers (2024-02-27T13:02:19Z) - Gemini vs GPT-4V: A Preliminary Comparison and Combination of
Vision-Language Models Through Qualitative Cases [98.35348038111508]
This paper presents an in-depth comparative study of two pioneering models: Google's Gemini and OpenAI's GPT-4V(ision)
The core of our analysis delves into the distinct visual comprehension abilities of each model.
Our findings illuminate the unique strengths and niches of both models.
arXiv Detail & Related papers (2023-12-22T18:59:58Z) - Large Language Models for Propaganda Span Annotation [11.64165958410489]
We investigate whether large language models (LLMs), such as GPT-4, can effectively perform the task.
We use a large-scale in-house dataset consisting of annotations from human annotators with varying expertise levels.
We plan to make the collected span-level labels from multiple annotators, including GPT-4, available for the community.
arXiv Detail & Related papers (2023-11-16T11:37:54Z) - Holistic Analysis of Hallucination in GPT-4V(ision): Bias and
Interference Challenges [54.42256219010956]
This benchmark is designed to evaluate and shed light on the two common types of hallucinations in visual language models: bias and interference.
bias refers to the model's tendency to hallucinate certain types of responses, possibly due to imbalance in its training data.
interference pertains to scenarios where the judgment of GPT-4V(ision) can be disrupted due to how the text prompt is phrased or how the input image is presented.
arXiv Detail & Related papers (2023-11-06T17:26:59Z) - The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision) [121.42924593374127]
We analyze the latest model, GPT-4V, to deepen the understanding of LMMs.
GPT-4V's unprecedented ability in processing arbitrarily interleaved multimodal inputs makes it a powerful multimodal generalist system.
GPT-4V's unique capability of understanding visual markers drawn on input images can give rise to new human-computer interaction methods.
arXiv Detail & Related papers (2023-09-29T17:34:51Z) - Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models [6.145834902689888]
Large language models (LLMs) have demonstrated impressive performance on various downstream tasks without requiring fine-tuning.
Despite having a lower training proportion compared to English, these models also exhibit remarkable capabilities in other languages.
In this study, we assess the performance of GPT-3.5 and GPT-4 models on seven distinct Arabic NLP tasks.
arXiv Detail & Related papers (2023-06-28T15:54:29Z) - GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot
Setting and Performance Boosting Through Prompts [0.0]
Large Language Models (LLMs) have exhibited remarkable performance on various Natural Language Processing (NLP) tasks.
In this paper, we examine the performance of GPT-3.5, GPT-4, and BARD models, by performing a thorough technical evaluation on different reasoning tasks.
arXiv Detail & Related papers (2023-05-21T14:45:17Z) - Document-Level Machine Translation with Large Language Models [91.03359121149595]
Large language models (LLMs) can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks.
This paper provides an in-depth evaluation of LLMs' ability on discourse modeling.
arXiv Detail & Related papers (2023-04-05T03:49:06Z) - Large Language Models in the Workplace: A Case Study on Prompt
Engineering for Job Type Classification [58.720142291102135]
This case study investigates the task of job classification in a real-world setting.
The goal is to determine whether an English-language job posting is appropriate for a graduate or entry-level position.
arXiv Detail & Related papers (2023-03-13T14:09:53Z) - Prompting GPT-3 To Be Reliable [117.23966502293796]
This work decomposes reliability into four facets: generalizability, fairness, calibration, and factuality.
We find that GPT-3 outperforms smaller-scale supervised models by large margins on all these facets.
arXiv Detail & Related papers (2022-10-17T14:52:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.