Large language models can rate news outlet credibility
- URL: http://arxiv.org/abs/2304.00228v1
- Date: Sat, 1 Apr 2023 05:04:06 GMT
- Title: Large language models can rate news outlet credibility
- Authors: Kai-Cheng Yang and Filippo Menczer
- Abstract summary: Large language models (LLMs) have shown exceptional performance in various natural language processing tasks.
Here we assess whether ChatGPT, a prominent LLM, can evaluate the credibility of news outlets.
Our results show that these ratings correlate with those from human experts.
- Score: 6.147741269183294
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Although large language models (LLMs) have shown exceptional performance in
various natural language processing tasks, they are prone to hallucinations.
State-of-the-art chatbots, such as the new Bing, attempt to mitigate this issue
by gathering information directly from the internet to ground their answers. In
this setting, the capacity to distinguish trustworthy sources is critical for
providing appropriate accuracy contexts to users. Here we assess whether
ChatGPT, a prominent LLM, can evaluate the credibility of news outlets. With
appropriate instructions, ChatGPT can provide ratings for a diverse set of news
outlets, including those in non-English languages and satirical sources, along
with contextual explanations. Our results show that these ratings correlate
with those from human experts (Spearmam's $\rho=0.54, p<0.001$). These findings
suggest that LLMs could be an affordable reference for credibility ratings in
fact-checking applications. Future LLMs should enhance their alignment with
human expert judgments of source credibility to improve information accuracy.
Related papers
- Seeing Through AI's Lens: Enhancing Human Skepticism Towards LLM-Generated Fake News [0.38233569758620056]
This paper aims to elucidate simple markers that help individuals distinguish between articles penned by humans and those created by LLMs.
We then devise a metric named Entropy-Shift Authorship Signature (ESAS) based on the information theory and entropy principles.
The proposed ESAS ranks terms or entities, like POS tagging, within news articles based on their relevance in discerning article authorship.
arXiv Detail & Related papers (2024-06-20T06:02:04Z) - Open Source Conversational LLMs do not know most Spanish words [2.737783055857426]
We evaluate the knowledge that open-source chat LLMs have of Spanish words by testing a sample of words in a reference dictionary.
Results show that open-source chat LLMs produce incorrect meanings for an important fraction of the words and are not able to use most of the words correctly to write sentences with context.
arXiv Detail & Related papers (2024-03-21T15:41:02Z) - Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation [64.5862977630713]
This study investigates how Large Language Models (LLMs) leverage source and reference data in machine translation evaluation task.
We find that reference information significantly enhances the evaluation accuracy, while surprisingly, source information sometimes is counterproductive.
arXiv Detail & Related papers (2024-01-12T13:23:21Z) - The Perils & Promises of Fact-checking with Large Language Models [55.869584426820715]
Large Language Models (LLMs) are increasingly trusted to write academic papers, lawsuits, and news articles.
We evaluate the use of LLM agents in fact-checking by having them phrase queries, retrieve contextual data, and make decisions.
Our results show the enhanced prowess of LLMs when equipped with contextual information.
While LLMs show promise in fact-checking, caution is essential due to inconsistent accuracy.
arXiv Detail & Related papers (2023-10-20T14:49:47Z) - Let Models Speak Ciphers: Multiagent Debate through Embeddings [84.20336971784495]
We introduce CIPHER (Communicative Inter-Model Protocol Through Embedding Representation) to address this issue.
By deviating from natural language, CIPHER offers an advantage of encoding a broader spectrum of information without any modification to the model weights.
This showcases the superiority and robustness of embeddings as an alternative "language" for communication among LLMs.
arXiv Detail & Related papers (2023-10-10T03:06:38Z) - Fake News Detectors are Biased against Texts Generated by Large Language
Models [39.36284616311687]
The spread of fake news has emerged as a critical challenge, undermining trust and posing threats to society.
We present a novel paradigm to evaluate fake news detectors in scenarios involving both human-written and LLM-generated misinformation.
arXiv Detail & Related papers (2023-09-15T18:04:40Z) - "According to ...": Prompting Language Models Improves Quoting from
Pre-Training Data [52.03853726206584]
Large Language Models (LLMs) may hallucinate and generate fake information, despite pre-training on factual data.
We propose according-to prompting: directing LLMs to ground responses against previously observed text.
To quantify this grounding, we propose a novel evaluation metric (QUIP-Score) that measures the extent to which model-produced answers are directly found in underlying text corpora.
arXiv Detail & Related papers (2023-05-22T17:25:24Z) - ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political
Twitter Messages with Zero-Shot Learning [0.0]
This paper assesses the accuracy, reliability and bias of the Large Language Model (LLM) ChatGPT-4 on the text analysis task of classifying the political affiliation of a Twitter poster based on the content of a tweet.
We use Twitter messages from United States politicians during the 2020 election, providing a ground truth against which to measure accuracy.
arXiv Detail & Related papers (2023-04-13T14:51:40Z) - Multiverse: Multilingual Evidence for Fake News Detection [71.51905606492376]
Multiverse is a new feature based on multilingual evidence that can be used for fake news detection.
The hypothesis of the usage of cross-lingual evidence as a feature for fake news detection is confirmed.
arXiv Detail & Related papers (2022-11-25T18:24:17Z) - Explainable Tsetlin Machine framework for fake news detection with
credibility score assessment [16.457778420360537]
We propose a novel interpretable fake news detection framework based on the recently introduced Tsetlin Machine (TM)
We use the conjunctive clauses of the TM to capture lexical and semantic properties of both true and fake news text.
For evaluation, we conduct experiments on two publicly available datasets, PolitiFact and GossipCop, and demonstrate that the TM framework significantly outperforms previously published baselines by at least $5%$ in terms of accuracy.
arXiv Detail & Related papers (2021-05-19T13:18:02Z) - Viable Threat on News Reading: Generating Biased News Using Natural
Language Models [49.90665530780664]
We show that publicly available language models can reliably generate biased news content based on an input original news.
We also show that a large number of high-quality biased news articles can be generated using controllable text generation.
arXiv Detail & Related papers (2020-10-05T16:55:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.