A Survey on Automatic Credibility Assessment of Textual Credibility Signals in the Era of Large Language Models

• URL: http://arxiv.org/abs/2410.21360v1
• Date: Mon, 28 Oct 2024 17:51:08 GMT
• Title: A Survey on Automatic Credibility Assessment of Textual Credibility Signals in the Era of Large Language Models
• Authors: Ivan Srba, Olesya Razuvayevskaya, João A. Leite, Robert Moro, Ipek Baris Schlicht, Sara Tonelli, Francisco Moreno García, Santiago Barrio Lottmann, Denis Teyssou, Valentin Porcellini, Carolina Scarton, Kalina Bontcheva, Maria Bielikova,
• Abstract summary: Credibility assessment is fundamentally based on aggregating credibility signals. Credibility signals provide a more granular, more easily explainable and widely utilizable information. A growing body of research on automatic credibility assessment and detection of credibility signals can be characterized as highly fragmented and lacking mutual interconnections.
• Score: 6.538395325419292
• License:
• Abstract: In the current era of social media and generative AI, an ability to automatically assess the credibility of online social media content is of tremendous importance. Credibility assessment is fundamentally based on aggregating credibility signals, which refer to small units of information, such as content factuality, bias, or a presence of persuasion techniques, into an overall credibility score. Credibility signals provide a more granular, more easily explainable and widely utilizable information in contrast to currently predominant fake news detection, which utilizes various (mostly latent) features. A growing body of research on automatic credibility assessment and detection of credibility signals can be characterized as highly fragmented and lacking mutual interconnections. This issue is even more prominent due to a lack of an up-to-date overview of research works on automatic credibility assessment. In this survey, we provide such systematic and comprehensive literature review of 175 research papers while focusing on textual credibility signals and Natural Language Processing (NLP), which undergoes a significant advancement due to Large Language Models (LLMs). While positioning the NLP research into the context of other multidisciplinary research works, we tackle with approaches for credibility assessment as well as with 9 categories of credibility signals (we provide a thorough analysis for 3 of them, namely: 1) factuality, subjectivity and bias, 2) persuasion techniques and logical fallacies, and 3) claims and veracity). Following the description of the existing methods, datasets and tools, we identify future challenges and opportunities, while paying a specific attention to recent rapid development of generative AI.

Related papers

• A Survey of Stance Detection on Social Media: New Directions and Perspectives [50.27382951812502]
  stance detection has emerged as a crucial subfield within affective computing. Recent years have seen a surge of research interest in developing effective stance detection methods. This paper provides a comprehensive survey of stance detection techniques on social media.
  arXiv  Detail & Related papers  (2024-09-24T03:06:25Z)
• RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance [0.8089605035945486]
  We propose RelevAI-Reviewer, an automatic system that conceptualizes the task of survey paper review as a classification problem. We introduce a novel dataset comprised of 25,164 instances. Each instance contains one prompt and four candidate papers, each varying in relevance to the prompt. We develop a machine learning (ML) model capable of determining the relevance of each paper and identifying the most pertinent one.
  arXiv  Detail & Related papers  (2024-06-13T06:42:32Z)
• Exploring the Potential of the Large Language Models (LLMs) in Identifying Misleading News Headlines [2.0330684186105805]
  This study explores the efficacy of Large Language Models (LLMs) in identifying misleading versus non-misleading news headlines. Our analysis reveals significant variance in model performance, with ChatGPT-4 demonstrating superior accuracy.
  arXiv  Detail & Related papers  (2024-05-06T04:06:45Z)
• Maintaining Journalistic Integrity in the Digital Age: A Comprehensive NLP Framework for Evaluating Online News Content [0.0]
  This paper proposes a comprehensive framework to analyze online news texts using natural language processing (NLP) techniques. The framework incorporates ten journalism standards-objectivity, balance and fairness, readability and clarity, sensationalism and clickbait, ethical considerations, public interest and value, source credibility, relevance and timeliness, factual accuracy, and attribution and transparency.
  arXiv  Detail & Related papers  (2024-01-07T12:27:14Z)
• A Study of Situational Reasoning for Traffic Understanding [63.45021731775964]
  We devise three novel text-based tasks for situational reasoning in the traffic domain. We adopt four knowledge-enhanced methods that have shown generalization capability across language reasoning tasks in prior work. We provide in-depth analyses of model performance on data partitions and examine model predictions categorically.
  arXiv  Detail & Related papers  (2023-06-05T01:01:12Z)
• Consistency Analysis of ChatGPT [65.268245109828]
  This paper investigates the trustworthiness of ChatGPT and GPT-4 regarding logically consistent behaviour. Our findings suggest that while both models appear to show an enhanced language understanding and reasoning ability, they still frequently fall short of generating logically consistent predictions.
  arXiv  Detail & Related papers  (2023-03-11T01:19:01Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Exploring the Trade-off between Plausibility, Change Intensity and Adversarial Power in Counterfactual Explanations using Multi-objective Optimization [73.89239820192894]
  We argue that automated counterfactual generation should regard several aspects of the produced adversarial instances. We present a novel framework for the generation of counterfactual examples.
  arXiv  Detail & Related papers  (2022-05-20T15:02:53Z)
• User Experience Design for Automatic Credibility Assessment of News Content About COVID-19 [0.33262200259340124]
  We present two empirical studies to evaluate the usability of graphical interfaces that offer credibility assessment. Rating scale, sub-criteria and algorithm authorship are important predictors of the usability. The authorship of a news text is more important than the authorship of the credibility algorithm used to assess the content quality.
  arXiv  Detail & Related papers  (2022-04-29T08:38:45Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)
• Information Credibility in the Social Web: Contexts, Approaches, and Open Issues [2.2133187119466116]
  Credibility, also referred as believability, is a quality perceived by individuals, who are not always able to discern, with their own cognitive capacities, genuine information from fake one. Several approaches have been proposed to automatically assess credibility in social media.
  arXiv  Detail & Related papers  (2020-01-26T15:42:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.