On the Effectiveness of Fact Checking Information from Politically Congruent and Incongruent Large Language Models

• URL: http://arxiv.org/abs/2607.15364v1
• Date: Thu, 16 Jul 2026 18:08:09 GMT
• Title: On the Effectiveness of Fact Checking Information from Politically Congruent and Incongruent Large Language Models
• Abstract summary: Social media companies have shifted away from human fact-checkers and instead have embedded conversational Large Language Models (LLM) on their platforms.<n>This study investigates the effectiveness of fact checking information from ideologically configured LLM chatbots.<n>We find that LLM fact-checkers significantly shift trust in true and false political news headlines, even when the bot is politically incongruent with the user.
• Score: 0.28675177318965045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Social media companies have shifted away from human fact-checkers and instead have embedded conversational Large Language Models (LLM) on their platforms. LLM chatbots differ from human fact-checkers in many ways that may shape user responses to corrections. Of particular interest in this study is that LLM chatbots can be ideologically configured via the content emphasized in their responses, the sources cited, and the configured persona. Using data from two within-subjects experiments (n=705), this paper investigates the effectiveness of fact checking information from ideologically configured LLM chatbots. We find that LLM fact-checkers significantly shift trust in true and false political news headlines, even when the chatbot is politically incongruent with the user. The perceived political congruency between the participant and the bot matters only when headlines are politically distant. That is, trust in correctly labeled true headlines increases less when politically distant chatbots check distant headlines and increases more when moderate chatbots check distant headlines. The perceived political congruency of LLM chatbots did not impact their effectiveness at decreasing trust in false headlines. Unfortunately, LLM fact-checkers also significantly change trust in news when they are wrong or provide inconclusive answers. Our results demonstrate both the potential for LLMs to correct false information at scale but also their potential to taint the truth at scale.

Related papers

• Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters [0.571853823214391]
  Large language models (LLMs) are predominantly used by many as a primary source of information for various topics.<n>LLMs frequently make factual errors, fabricate data (hallucinations), or present biases, exposing users to misinformation and influencing opinions.<n>We quantify the political bias of popular LLMs in the context of the recent vote of the German Bundestag using the score produced by the Wahl-O-Mat.
  arXiv  Detail & Related papers  (2025-05-07T13:18:41Z)
• Large Language Models are often politically extreme, usually ideologically inconsistent, and persuasive even in informational contexts [1.9782163071901029]
  Large Language Models (LLMs) are a transformational technology, fundamentally changing how people obtain information and interact with the world.<n>We show that LLMs' apparently small overall partisan preference is the net result of offsetting extreme views on specific topics.<n>In a randomized experiment, we show that LLMs can promulgate their preferences into political persuasiveness even in information-seeking contexts.
  arXiv  Detail & Related papers  (2025-05-07T06:53:59Z)
• "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust [51.542856739181474]
  We show how different natural language expressions of uncertainty impact participants' reliance, trust, and overall task performance. We find that first-person expressions decrease participants' confidence in the system and tendency to agree with the system's answers, while increasing participants' accuracy. Our findings suggest that using natural language expressions of uncertainty may be an effective approach for reducing overreliance on LLMs, but that the precise language used matters.
  arXiv  Detail & Related papers  (2024-05-01T16:43:55Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference [48.99117537559644]
  We introduce Arena, an open platform for evaluating Large Language Models (LLMs) based on human preferences. Our methodology employs a pairwise comparison approach and leverages input from a diverse user base through crowdsourcing. This paper describes the platform, analyzes the data we have collected so far, and explains the tried-and-true statistical methods we are using.
  arXiv  Detail & Related papers  (2024-03-07T01:22:38Z)
• The Earth is Flat? Unveiling Factual Errors in Large Language Models [89.94270049334479]
  Large Language Models (LLMs) like ChatGPT are in various applications due to their extensive knowledge from pre-training and fine-tuning. Despite this, they are prone to generating factual and commonsense errors, raising concerns in critical areas like healthcare, journalism, and education. We introduce a novel, automatic testing framework, FactChecker, aimed at uncovering factual inaccuracies in LLMs.
  arXiv  Detail & Related papers  (2024-01-01T14:02:27Z)
• The Perils & Promises of Fact-checking with Large Language Models [55.869584426820715]
  Large Language Models (LLMs) are increasingly trusted to write academic papers, lawsuits, and news articles. We evaluate the use of LLM agents in fact-checking by having them phrase queries, retrieve contextual data, and make decisions. Our results show the enhanced prowess of LLMs when equipped with contextual information. While LLMs show promise in fact-checking, caution is essential due to inconsistent accuracy.
  arXiv  Detail & Related papers  (2023-10-20T14:49:47Z)
• Fact-checking information from large language models can decrease headline discernment [6.814801748069122]
  We investigate the impact of fact-checking information generated by a popular large language model on belief in, and sharing intent of, political news headlines. We find that this information does not significantly improve participants' ability to discern headline accuracy or share accurate news. Our findings highlight an important source of potential harm stemming from AI applications.
  arXiv  Detail & Related papers  (2023-08-21T15:47:37Z)
• ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning [0.0]
  This paper assesses the accuracy, reliability and bias of the Large Language Model (LLM) ChatGPT-4 on the text analysis task of classifying the political affiliation of a Twitter poster based on the content of a tweet. We use Twitter messages from United States politicians during the 2020 election, providing a ground truth against which to measure accuracy.
  arXiv  Detail & Related papers  (2023-04-13T14:51:40Z)
• Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models [8.367075755850983]
  This paper audits nine widely used language models (LLMs) from three leading providers to evaluate their ability to discern credible and high-quality information sources.<n>We find that larger models more frequently refuse to provide ratings due to insufficient information, whereas smaller models are more prone to making errors in their ratings.
  arXiv  Detail & Related papers  (2023-04-01T05:04:06Z)
• Right and left, partisanship predicts (asymmetric) vulnerability to misinformation [71.46564239895892]
  We analyze the relationship between partisanship, echo chambers, and vulnerability to online misinformation by studying news sharing behavior on Twitter. We find that vulnerability to misinformation is most strongly influenced by partisanship for both left- and right-leaning users.
  arXiv  Detail & Related papers  (2020-10-04T01:36:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.