Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals

• URL: http://arxiv.org/abs/2309.07601v3
• Date: Mon, 04 Nov 2024 21:33:00 GMT
• Title: Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals
• Authors: João A. Leite, Olesya Razuvayevskaya, Kalina Bontcheva, Carolina Scarton,
• Abstract summary: Pastel is a weakly supervised approach that leverages large language models to extract credibility signals from web content. We study the association between credibility signals and veracity, and perform a study showing the impact of each signal on model performance.
• Score: 4.895830603263421
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Credibility signals represent a wide range of heuristics typically used by journalists and fact-checkers to assess the veracity of online content. Automating the extraction of credibility signals presents significant challenges due to the necessity of training high-accuracy, signal-specific extractors, coupled with the lack of sufficiently large annotated datasets. This paper introduces Pastel (Prompted weAk Supervision wiTh crEdibility signaLs), a weakly supervised approach that leverages large language models (LLMs) to extract credibility signals from web content, and subsequently combines them to predict the veracity of content without relying on human supervision. We validate our approach using four article-level misinformation detection datasets, demonstrating that Pastel outperforms zero-shot veracity detection by 38.3% and achieves 86.7% of the performance of the state-of-the-art system trained with human supervision. Moreover, in cross-domain settings where training and testing datasets originate from different domains, Pastel significantly outperforms the state-of-the-art supervised model by 63%. We further study the association between credibility signals and veracity, and perform an ablation study showing the impact of each signal on model performance. Our findings reveal that 12 out of the 19 proposed signals exhibit strong associations with veracity across all datasets, while some signals show domain-specific strengths.

Related papers

• Learning High-Quality Latent Representations for Anomaly Detection and Signal Integrity Enhancement in High-Speed Signals [3.0017241250121387]
  This paper addresses the dual challenge of improving anomaly detection and signal integrity in high-speed dynamic random access memory signals.<n>We propose a joint training framework that integrates an autoencoder with a classifier to learn more distinctive latent representations.<n>We introduce a signal integrity enhancement algorithm that improves signal integrity by an average of 11.3%.
  arXiv  Detail & Related papers  (2025-06-23T04:48:22Z)
• SelfCheckAgent: Zero-Resource Hallucination Detection in Generative Large Language Models [0.16385815610837165]
  SelfCheckAgent is a novel framework integrating three different agents. These agents provide a robust multi-dimensional approach to hallucination detection. The framework also incorporates a triangulation strategy, which increases the strengths of the SelfCheckAgent.
  arXiv  Detail & Related papers  (2025-02-03T20:42:32Z)
• Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Study [61.65123150513683]
  multimodal foundation models, such as CLIP, produce state-of-the-art zero-shot results. It is reported that these models close the robustness gap by matching the performance of supervised models trained on ImageNet. We show that CLIP leads to a significant robustness drop compared to supervised ImageNet models on our benchmark.
  arXiv  Detail & Related papers  (2024-03-15T17:33:49Z)
• Always be Pre-Training: Representation Learning for Network Intrusion Detection with GNNs [6.589041710104928]
  Graph neural network-based network intrusion detection systems have recently demonstrated state-of-the-art performance on benchmark datasets. These methods suffer from a reliance on target encoding for data pre-processing, limiting widespread adoption due to the associated need for annotated labels. We propose a solution involving in-context pre-training and the utilization of dense representations for categorical features to jointly overcome the label-dependency limitation.
  arXiv  Detail & Related papers  (2024-02-29T09:40:07Z)
• The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness [50.52507648690234]
  Federated learning has the risk of skewing fine-tuning features and compromising the robustness of the model. We introduce three robustness indicators and conduct experiments across diverse robust datasets. Our approach markedly enhances the robustness across diverse scenarios, encompassing various parameter-efficient fine-tuning methods.
  arXiv  Detail & Related papers  (2024-01-25T09:18:51Z)
• Fusing Pseudo Labels with Weak Supervision for Dynamic Traffic Scenarios [0.0]
  We introduce a weakly-supervised label unification pipeline that amalgamates pseudo labels from object detection models trained on heterogeneous datasets. Our pipeline engenders a unified label space through the amalgamation of labels from disparate datasets, rectifying bias and enhancing generalization. We retrain a solitary object detection model using the merged label space, culminating in a resilient model proficient in dynamic traffic scenarios.
  arXiv  Detail & Related papers  (2023-08-30T11:33:07Z)
• Data AUDIT: Identifying Attribute Utility- and Detectability-Induced Bias in Task Models [8.420252576694583]
  We present a first technique for the rigorous, quantitative screening of medical image datasets. Our method decomposes the risks associated with dataset attributes in terms of their detectability and utility. Using our method, we show our screening method reliably identifies nearly imperceptible bias-inducing artifacts.
  arXiv  Detail & Related papers  (2023-04-06T16:50:15Z)
• Cluster-level pseudo-labelling for source-free cross-domain facial expression recognition [94.56304526014875]
  We propose the first Source-Free Unsupervised Domain Adaptation (SFUDA) method for Facial Expression Recognition (FER) Our method exploits self-supervised pretraining to learn good feature representations from the target data. We validate the effectiveness of our method in four adaptation setups, proving that it consistently outperforms existing SFUDA methods when applied to FER.
  arXiv  Detail & Related papers  (2022-10-11T08:24:50Z)
• Generative Modeling Helps Weak Supervision (and Vice Versa) [87.62271390571837]
  We propose a model fusing weak supervision and generative adversarial networks. It captures discrete variables in the data alongside the weak supervision derived label estimate. It is the first approach to enable data augmentation through weakly supervised synthetic images and pseudolabels.
  arXiv  Detail & Related papers  (2022-03-22T20:24:21Z)
• Towards Reducing Labeling Cost in Deep Object Detection [61.010693873330446]
  We propose a unified framework for active learning, that considers both the uncertainty and the robustness of the detector. Our method is able to pseudo-label the very confident predictions, suppressing a potential distribution drift.
  arXiv  Detail & Related papers  (2021-06-22T16:53:09Z)
• ReLearn: A Robust Machine Learning Framework in Presence of Missing Data for Multimodal Stress Detection from Physiological Signals [5.042598205771715]
  We propose ReLearn, a robust machine learning framework for stress detection from biomarkers extracted from multimodal physiological signals. ReLearn effectively copes with missing data and outliers both at training and inference phases. Our experiments show that the proposed framework obtains a cross-validation accuracy of 86.8% even if more than 50% of samples within the features are missing.
  arXiv  Detail & Related papers  (2021-04-29T11:53:01Z)
• Negative Data Augmentation [127.28042046152954]
  We show that negative data augmentation samples provide information on the support of the data distribution. We introduce a new GAN training objective where we use NDA as an additional source of synthetic data for the discriminator. Empirically, models trained with our method achieve improved conditional/unconditional image generation along with improved anomaly detection capabilities.
  arXiv  Detail & Related papers  (2021-02-09T20:28:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.