Large Language Models for Propaganda Detection

• URL: http://arxiv.org/abs/2310.06422v2
• Date: Mon, 27 Nov 2023 10:18:36 GMT
• Title: Large Language Models for Propaganda Detection
• Authors: Kilian Sprenkamp, Daniel Gordon Jones, Liudmila Zavolokina
• Abstract summary: This study investigates the effectiveness of Large Language Models (LLMs) for propaganda detection. Five variations of GPT-3 and GPT-4 are employed, incorporating various prompt engineering and fine-tuning strategies. Our findings demonstrate that GPT-4 achieves comparable results to the current state-of-the-art.
• Score: 2.587450057509126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The prevalence of propaganda in our digital society poses a challenge to societal harmony and the dissemination of truth. Detecting propaganda through NLP in text is challenging due to subtle manipulation techniques and contextual dependencies. To address this issue, we investigate the effectiveness of modern Large Language Models (LLMs) such as GPT-3 and GPT-4 for propaganda detection. We conduct experiments using the SemEval-2020 task 11 dataset, which features news articles labeled with 14 propaganda techniques as a multi-label classification problem. Five variations of GPT-3 and GPT-4 are employed, incorporating various prompt engineering and fine-tuning strategies across the different models. We evaluate the models' performance by assessing metrics such as $F1$ score, $Precision$, and $Recall$, comparing the results with the current state-of-the-art approach using RoBERTa. Our findings demonstrate that GPT-4 achieves comparable results to the current state-of-the-art. Further, this study analyzes the potential and challenges of LLMs in complex tasks like propaganda detection.

Related papers

• PropaInsight: Toward Deeper Understanding of Propaganda in Terms of Techniques, Appeals, and Intent [71.20471076045916]
  Propaganda plays a critical role in shaping public opinion and fueling disinformation. Propainsight systematically dissects propaganda into techniques, arousal appeals, and underlying intent. Propagaze combines human-annotated data with high-quality synthetic data.
  arXiv  Detail & Related papers  (2024-09-19T06:28:18Z)
• Can GPT-4 learn to analyse moves in research article abstracts? [0.9999629695552195]
  We employ the affordances of GPT-4 to automate the annotation process by using natural language prompts. An 8-shot prompt was more effective than one using two, confirming that the inclusion of examples illustrating areas of variability can enhance GPT-4's ability to recognize multiple moves in a single sentence.
  arXiv  Detail & Related papers  (2024-07-22T13:14:27Z)
• Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda Spans in News Articles [11.64165958410489]
  We develop the largest propaganda dataset to date, comprised of 8K paragraphs from newspaper articles, labeled at the text span level following a taxonomy of 23 propagandistic techniques. Our work offers the first attempt to understand the performance of large language models (LLMs), using GPT-4, for fine-grained propaganda detection from text. Results showed that GPT-4's performance degrades as the task moves from simply classifying a paragraph as propagandistic or not, to the fine-grained task of detecting propaganda techniques and their manifestation in text.
  arXiv  Detail & Related papers  (2024-02-27T13:02:19Z)
• Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases [98.35348038111508]
  This paper presents an in-depth comparative study of two pioneering models: Google's Gemini and OpenAI's GPT-4V(ision) The core of our analysis delves into the distinct visual comprehension abilities of each model. Our findings illuminate the unique strengths and niches of both models.
  arXiv  Detail & Related papers  (2023-12-22T18:59:58Z)
• Large Language Models for Propaganda Span Annotation [10.358271919023903]
  This study investigates whether Large Language Models, such as GPT-4, can effectively extract propagandistic spans. The experiments are performed over a large-scale in-house manually annotated dataset.
  arXiv  Detail & Related papers  (2023-11-16T11:37:54Z)
• Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges [54.42256219010956]
  This benchmark is designed to evaluate and shed light on the two common types of hallucinations in visual language models: bias and interference. bias refers to the model's tendency to hallucinate certain types of responses, possibly due to imbalance in its training data. interference pertains to scenarios where the judgment of GPT-4V(ision) can be disrupted due to how the text prompt is phrased or how the input image is presented.
  arXiv  Detail & Related papers  (2023-11-06T17:26:59Z)
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision) [121.42924593374127]
  We analyze the latest model, GPT-4V, to deepen the understanding of LMMs. GPT-4V's unprecedented ability in processing arbitrarily interleaved multimodal inputs makes it a powerful multimodal generalist system. GPT-4V's unique capability of understanding visual markers drawn on input images can give rise to new human-computer interaction methods.
  arXiv  Detail & Related papers  (2023-09-29T17:34:51Z)
• GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts [0.0]
  Large Language Models (LLMs) have exhibited remarkable performance on various Natural Language Processing (NLP) tasks. In this paper, we examine the performance of GPT-3.5, GPT-4, and BARD models, by performing a thorough technical evaluation on different reasoning tasks.
  arXiv  Detail & Related papers  (2023-05-21T14:45:17Z)
• Document-Level Machine Translation with Large Language Models [91.03359121149595]
  Large language models (LLMs) can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. This paper provides an in-depth evaluation of LLMs' ability on discourse modeling.
  arXiv  Detail & Related papers  (2023-04-05T03:49:06Z)
• Prompting GPT-3 To Be Reliable [117.23966502293796]
  This work decomposes reliability into four facets: generalizability, fairness, calibration, and factuality. We find that GPT-3 outperforms smaller-scale supervised models by large margins on all these facets.
  arXiv  Detail & Related papers  (2022-10-17T14:52:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.