"Whose Side Are You On?" Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selection

• URL: http://arxiv.org/abs/2503.20797v1
• Date: Sun, 23 Mar 2025 02:32:25 GMT
• Title: "Whose Side Are You On?" Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selection
• Authors: Muhammad Haroon, Magdalena Wojcieszak, Anshuman Chhabra,
• Abstract summary: Existing approaches to classifying ideology are limited in that they require extensive human effort, the labeling of large datasets, and are not able to adapt to evolving ideological contexts.<n>This paper explores the potential of Large Language Models for classifying the political ideology of online content in the context of the two-party US political spectrum through in-context learning.
• Score: 5.277598111323805
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid growth of social media platforms has led to concerns about radicalization, filter bubbles, and content bias. Existing approaches to classifying ideology are limited in that they require extensive human effort, the labeling of large datasets, and are not able to adapt to evolving ideological contexts. This paper explores the potential of Large Language Models (LLMs) for classifying the political ideology of online content in the context of the two-party US political spectrum through in-context learning (ICL). Our extensive experiments involving demonstration selection in label-balanced fashion, conducted on three datasets comprising news articles and YouTube videos, reveal that our approach significantly outperforms zero-shot and traditional supervised methods. Additionally, we evaluate the influence of metadata (e.g., content source and descriptions) on ideological classification and discuss its implications. Finally, we show how providing the source for political and non-political content influences the LLM's classification.

Related papers

• Probing the Subtle Ideological Manipulation of Large Language Models [0.3745329282477067]
  Large Language Models (LLMs) have transformed natural language processing, but concerns have emerged about their susceptibility to ideological manipulation. We introduce a novel multi-task dataset designed to reflect diverse ideological positions through tasks such as ideological QA, statement ranking, manifesto cloze completion, and Congress bill comprehension. Our findings indicate that fine-tuning significantly enhances nuanced ideological alignment, while explicit prompts provide only minor refinements.
  arXiv  Detail & Related papers  (2025-04-19T13:11:50Z)
• Talking Point based Ideological Discourse Analysis in News Events [62.18747509565779]
  We propose a framework motivated by the theory of ideological discourse analysis to analyze news articles related to real-world events. Our framework represents the news articles using a relational structure - talking points, which captures the interaction between entities, their roles, and media frames along with a topic of discussion. We evaluate our framework's ability to generate these perspectives through automated tasks - ideology and partisan classification tasks, supplemented by human validation.
  arXiv  Detail & Related papers  (2025-04-10T02:52:34Z)
• Navigating Nuance: In Quest for Political Truth [1.4127714091330967]
  We evaluate the performance of the Llama-3 (70B) language model on the Media Bias Identification Benchmark (MBIB)<n>Our findings underscore the challenges of detecting political bias and highlight the potential of transfer learning methods to enhance future models.
  arXiv  Detail & Related papers  (2025-01-01T09:24:47Z)
• Political-LLM: Large Language Models in Political Science [159.95299889946637]
  Large language models (LLMs) have been widely adopted in political science tasks.<n>Political-LLM aims to advance the comprehensive understanding of integrating LLMs into computational political science.
  arXiv  Detail & Related papers  (2024-12-09T08:47:50Z)
• Representation Bias in Political Sample Simulations with Large Language Models [54.48283690603358]
  This study seeks to identify and quantify biases in simulating political samples with Large Language Models. Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao dataset, and China Family Panel Studies.
  arXiv  Detail & Related papers  (2024-07-16T05:52:26Z)
• Political Leaning Inference through Plurinational Scenarios [4.899818550820576]
  This work focuses on three diverse regions in Spain (Basque Country, Catalonia and Galicia) to explore various methods for multi-party categorization. We use a two-step method involving unsupervised user representations obtained from the retweets and their subsequent use for political leaning detection.
  arXiv  Detail & Related papers  (2024-06-12T07:42:12Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• Modeling Political Orientation of Social Media Posts: An Extended Analysis [0.0]
  Developing machine learning models to characterize political polarization on online social media presents significant challenges. These challenges mainly stem from various factors such as the lack of annotated data, presence of noise in social media datasets, and the sheer volume of data. We introduce two methods that leverage on news media bias and post content to label social media posts. We demonstrate that current machine learning models can exhibit improved performance in predicting political orientation of social media posts.
  arXiv  Detail & Related papers  (2023-11-21T03:34:20Z)
• Inducing Political Bias Allows Language Models Anticipate Partisan Reactions to Controversies [5.958974943807783]
  This study addresses the challenge of understanding political bias in digitized discourse using Large Language Models (LLMs) We present a comprehensive analytical framework, consisting of Partisan Bias Divergence Assessment and Partisan Class Tendency Prediction. Our findings reveal the model's effectiveness in capturing emotional and moral nuances, albeit with some challenges in stance detection.
  arXiv  Detail & Related papers  (2023-11-16T08:57:53Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)
• The Face of Populism: Examining Differences in Facial Emotional Expressions of Political Leaders Using Machine Learning [50.24983453990065]
  We use a deep-learning approach to process a sample of 220 YouTube videos of political leaders from 15 different countries. We observe statistically significant differences in the average score of negative emotions between groups of leaders with varying degrees of populist rhetoric.
  arXiv  Detail & Related papers  (2023-04-19T18:32:49Z)
• Learning Ideological Embeddings from Information Cascades [11.898833102736255]
  We propose a model to learn the ideological leaning of each user in a multidimensional ideological space. Our model is able to learn the political stance of the social media users in a multidimensional ideological space.
  arXiv  Detail & Related papers  (2021-09-28T09:58:02Z)
• Reaching the bubble may not be enough: news media role in online political polarization [58.720142291102135]
  A way of reducing polarization would be by distributing cross-partisan news among individuals with distinct political orientations. This study investigates whether this holds in the context of nationwide elections in Brazil and Canada.
  arXiv  Detail & Related papers  (2021-09-18T11:34:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.