Can Large Language Models Understand Content and Propagation for
Misinformation Detection: An Empirical Study
- URL: http://arxiv.org/abs/2311.12699v1
- Date: Tue, 21 Nov 2023 16:03:51 GMT
- Title: Can Large Language Models Understand Content and Propagation for
Misinformation Detection: An Empirical Study
- Authors: Mengyang Chen, Lingwei Wei, Han Cao, Wei Zhou, Songlin Hu
- Abstract summary: Large Language Models (LLMs) have garnered significant attention for their powerful ability in natural language understanding and reasoning.
We present a comprehensive empirical study to explore the performance of LLMs on misinformation detection tasks.
- Score: 26.023148371263012
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Large Language Models (LLMs) have garnered significant attention for their
powerful ability in natural language understanding and reasoning. In this
paper, we present a comprehensive empirical study to explore the performance of
LLMs on misinformation detection tasks. This study stands as the pioneering
investigation into the understanding capabilities of multiple LLMs regarding
both content and propagation across social media platforms. Our empirical
studies on five misinformation detection datasets show that LLMs with diverse
prompts achieve comparable performance in text-based misinformation detection
but exhibit notably constrained capabilities in comprehending propagation
structure compared to existing models in propagation-based misinformation
detection. Besides, we further design four instruction-tuned strategies to
enhance LLMs for both content and propagation-based misinformation detection.
These strategies boost LLMs to actively learn effective features from multiple
instances or hard instances, and eliminate irrelevant propagation structures,
thereby achieving better detection performance. Extensive experiments further
demonstrate LLMs would play a better capacity in content and propagation
structure under these proposed strategies and achieve promising detection
performance. These findings highlight the potential ability of LLMs to detect
misinformation.
Related papers
- Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning [15.919493497867567]
This study aims to evaluate the performance of Multimodal Large Language Models (MLLMs) on the VALSE benchmark.
We conducted a comprehensive assessment of state-of-the-art MLLMs, varying in model size and pretraining datasets.
arXiv Detail & Related papers (2024-07-17T11:26:47Z) - Learning on Graphs with Large Language Models(LLMs): A Deep Dive into Model Robustness [39.57155321515097]
Large Language Models (LLMs) have demonstrated remarkable performance across various natural language processing tasks.
It remains unclear whether LLMs exhibit robustness in learning on graphs.
arXiv Detail & Related papers (2024-07-16T09:05:31Z) - CausalBench: A Comprehensive Benchmark for Causal Learning Capability of Large Language Models [27.362012903540492]
Causality reveals fundamental principles behind data distributions in real-world scenarios.
Large language models (LLMs) can understand causality directly impacts their efficacy across explaining outputs, adapting to new evidence, and generating counterfactuals.
This paper proposes a comprehensive benchmark, namely CausalBench, to evaluate the causality understanding capabilities of LLMs.
arXiv Detail & Related papers (2024-04-09T14:40:08Z) - MMIDR: Teaching Large Language Model to Interpret Multimodal Misinformation via Knowledge Distillation [15.343028838291078]
We propose MMIDR, a framework designed to teach LLMs in providing fluent and high-quality textual explanations for their decision-making process of multimodal misinformation.
To convert multimodal misinformation into an appropriate instruction-following format, we present a data augmentation perspective and pipeline.
Furthermore, we design an efficient knowledge distillation approach to distill the capability of proprietary LLMs in explaining multimodal misinformation into open-source LLMs.
arXiv Detail & Related papers (2024-03-21T06:47:28Z) - Characterizing Truthfulness in Large Language Model Generations with
Local Intrinsic Dimension [63.330262740414646]
We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs)
We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
arXiv Detail & Related papers (2024-02-28T04:56:21Z) - LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges.
Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model.
This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
arXiv Detail & Related papers (2024-02-26T07:33:05Z) - LEMMA: Towards LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation [58.524237916836164]
We propose LEMMA: LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation.
Our method improves the accuracy over the top baseline LVLM by 7% and 13% on Twitter and Fakeddit datasets respectively.
arXiv Detail & Related papers (2024-02-19T08:32:27Z) - C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations.
Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
arXiv Detail & Related papers (2024-02-17T11:28:08Z) - Exploring the Cognitive Knowledge Structure of Large Language Models: An
Educational Diagnostic Assessment Approach [50.125704610228254]
Large Language Models (LLMs) have not only exhibited exceptional performance across various tasks, but also demonstrated sparks of intelligence.
Recent studies have focused on assessing their capabilities on human exams and revealed their impressive competence in different domains.
We conduct an evaluation using MoocRadar, a meticulously annotated human test dataset based on Bloom taxonomy.
arXiv Detail & Related papers (2023-10-12T09:55:45Z) - Survey on Factuality in Large Language Models: Knowledge, Retrieval and
Domain-Specificity [61.54815512469125]
This survey addresses the crucial issue of factuality in Large Language Models (LLMs)
As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital.
arXiv Detail & Related papers (2023-10-11T14:18:03Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.