Positioning Political Texts with Large Language Models by Asking and Averaging

• URL: http://arxiv.org/abs/2311.16639v3
• Date: Thu, 5 Sep 2024 16:17:20 GMT
• Title: Positioning Political Texts with Large Language Models by Asking and Averaging
• Authors: Gaël Le Mens, Aina Gallego,
• Abstract summary: We ask an LLM where a tweet or a sentence of a political text stands on the focal dimension and take the average of the LLM responses to position political actors. The correlations between the position estimates obtained with the best LLMs and benchmarks based on text coding by experts, crowdworkers, or roll call votes exceed.90. Using instruction-tuned LLMs to position texts in policy and ideological spaces is fast, cost-efficient, reliable, and reproducible (in the case of open LLMs) even if the texts are short and written in different languages.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We use instruction-tuned Large Language Models (LLMs) like GPT-4, Llama 3, MiXtral, or Aya to position political texts within policy and ideological spaces. We ask an LLM where a tweet or a sentence of a political text stands on the focal dimension and take the average of the LLM responses to position political actors such as US Senators, or longer texts such as UK party manifestos or EU policy speeches given in 10 different languages. The correlations between the position estimates obtained with the best LLMs and benchmarks based on text coding by experts, crowdworkers, or roll call votes exceed .90. This approach is generally more accurate than the positions obtained with supervised classifiers trained on large amounts of research data. Using instruction-tuned LLMs to position texts in policy and ideological spaces is fast, cost-efficient, reliable, and reproducible (in the case of open LLMs) even if the texts are short and written in different languages. We conclude with cautionary notes about the need for empirical validation.

Related papers

• Large Language Models Reflect the Ideology of their Creators [73.25935570218375]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language. We uncover notable diversity in the ideological stance exhibited across different LLMs and languages.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Large Language Models' Detection of Political Orientation in Newspapers [0.0]
  Various methods have been developed to better understand newspapers' positioning. The advent of Large Language Models (LLM) hold disruptive potential to assist researchers and citizens alike. We compare how four widely employed LLMs rate the positioning of newspapers, and compare if their answers align with one another. Over a woldwide dataset, articles in newspapers are positioned strikingly differently by single LLMs, hinting to inconsistent training or excessive randomness in the algorithms.
  arXiv  Detail & Related papers  (2024-05-23T06:18:03Z)
• Assessing Political Bias in Large Language Models [0.624709220163167]
  We evaluate the political bias of open-source Large Language Models (LLMs) concerning political issues within the European Union (EU) from a German voter's perspective. We show that larger models, such as Llama3-70B, tend to align more closely with left-leaning political parties, while smaller models often remain neutral.
  arXiv  Detail & Related papers  (2024-05-17T15:30:18Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• Large Language Models: A Survey [69.72787936480394]
  Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data.
  arXiv  Detail & Related papers  (2024-02-09T05:37:09Z)
• The Political Preferences of LLMs [0.0]
  I administer 11 political orientation tests, designed to identify the political preferences of the test taker, to 24 state-of-the-art conversational LLMs. Most conversational LLMs generate responses that are diagnosed by most political test instruments as manifesting preferences for left-of-center viewpoints. I demonstrate that LLMs can be steered towards specific locations in the political spectrum through Supervised Fine-Tuning.
  arXiv  Detail & Related papers  (2024-02-02T02:43:10Z)
• Measurement in the Age of LLMs: An Application to Ideological Scaling [1.9413548770753526]
  This paper explores the use of large language models (LLMs) to navigate the conceptual clutter inherent to social scientific measurement tasks. We rely on LLMs' remarkable linguistic fluency to elicit ideological scales of both legislators and text.
  arXiv  Detail & Related papers  (2023-12-14T18:34:06Z)
• AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations [52.43593893122206]
  Alignedcot is an in-context learning technique for invoking Large Language Models. It achieves consistent and correct step-wise prompts in zero-shot scenarios. We conduct experiments on mathematical reasoning and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-11-22T17:24:21Z)
• Towards Human-Level Text Coding with LLMs: The Case of Fatherhood Roles in Public Policy Documents [19.65846717628022]
  Large language models (LLMs) promise automation with better results and less programming. In this study, we evaluate LLMs on three original coding tasks involving typical complexities encountered in political science settings. We find that the best prompting strategy consists of providing the LLMs with a detailed codebook, as the one provided to human coders.
  arXiv  Detail & Related papers  (2023-11-20T15:34:45Z)
• LM-Polygraph: Uncertainty Estimation for Language Models [71.21409522341482]
  Uncertainty estimation (UE) methods are one path to safer, more responsible, and more effective use of large language models (LLMs) We introduce LM-Polygraph, a framework with implementations of a battery of state-of-the-art UE methods for LLMs in text generation tasks, with unified program interfaces in Python. It introduces an extendable benchmark for consistent evaluation of UE techniques by researchers, and a demo web application that enriches the standard chat dialog with confidence scores.
  arXiv  Detail & Related papers  (2023-11-13T15:08:59Z)
• Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs [59.74002011562726]
  We propose a novel linguistic cue-based chain-of-thoughts (textitCue-CoT) to provide a more personalized and engaging response. We build a benchmark with in-depth dialogue questions, consisting of 6 datasets in both Chinese and English. Empirical results demonstrate our proposed textitCue-CoT method outperforms standard prompting methods in terms of both textithelpfulness and textitacceptability on all datasets.
  arXiv  Detail & Related papers  (2023-05-19T16:27:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.