Ta'keed: The First Generative Fact-Checking System for Arabic Claims

• URL: http://arxiv.org/abs/2401.14067v1
• Date: Thu, 25 Jan 2024 10:43:00 GMT
• Title: Ta'keed: The First Generative Fact-Checking System for Arabic Claims
• Authors: Saud Althabiti, Mohammad Ammar Alsalka, and Eric Atwell
• Abstract summary: This paper introduces Ta'keed, an explainable Arabic automatic fact-checking system. Ta'keed generates explanations for claim credibility, particularly in Arabic. The system achieved a promising F1 score of 0.72 in the classification task.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces Ta'keed, an explainable Arabic automatic fact-checking system. While existing research often focuses on classifying claims as "True" or "False," there is a limited exploration of generating explanations for claim credibility, particularly in Arabic. Ta'keed addresses this gap by assessing claim truthfulness based on retrieved snippets, utilizing two main components: information retrieval and LLM-based claim verification. We compiled the ArFactEx, a testing gold-labelled dataset with manually justified references, to evaluate the system. The initial model achieved a promising F1 score of 0.72 in the classification task. Meanwhile, the system's generated explanations are compared with gold-standard explanations syntactically and semantically. The study recommends evaluating using semantic similarities, resulting in an average cosine similarity score of 0.76. Additionally, we explored the impact of varying snippet quantities on claim classification accuracy, revealing a potential correlation, with the model using the top seven hits outperforming others with an F1 score of 0.77.

Related papers

• Localizing Factual Inconsistencies in Attributable Text Generation [91.981439746404]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation. We first demonstrate the effectiveness of the QASemConsistency methodology for human annotation. We then implement several methods for automatically detecting localized factual inconsistencies.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• Evaluating Evidence Attribution in Generated Fact Checking Explanations [48.776087871960584]
  We introduce a novel evaluation protocol, citation masking and recovery, to assess attribution quality in generated explanations. Experiments reveal that the best-performing LLMs still generate explanations with inaccurate attributions. Human-curated evidence is essential for generating better explanations.
  arXiv  Detail & Related papers  (2024-06-18T14:13:13Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models [55.60306377044225]
  "SelfCheckGPT" is a simple sampling-based approach to fact-check the responses of black-box models. We investigate this approach by using GPT-3 to generate passages about individuals from the WikiBio dataset.
  arXiv  Detail & Related papers  (2023-03-15T19:31:21Z)
• Aggregating Pairwise Semantic Differences for Few-Shot Claim Veracity Classification [21.842139093124512]
  We introduce SEED, a novel vector-based method to claim veracity classification. We build on the hypothesis that we can simulate class representative vectors that capture average semantic differences for claim-evidence pairs in a class. Experiments conducted on the FEVER and SCIFACT datasets show consistent improvements over competitive baselines in few-shot settings.
  arXiv  Detail & Related papers  (2022-05-11T17:23:37Z)
• Overview of the CLEF-2019 CheckThat!: Automatic Identification and Verification of Claims [26.96108180116284]
  CheckThat! lab featured two tasks in two different languages: English and Arabic. The most successful approaches to Task 1 used various neural networks and logistic regression. Learning-to-rank was used by the highest scoring runs for subtask A.
  arXiv  Detail & Related papers  (2021-09-25T16:08:09Z)
• Visualizing Classifier Adjacency Relations: A Case Study in Speaker Verification and Voice Anti-Spoofing [72.4445825335561]
  We propose a simple method to derive 2D representation from detection scores produced by an arbitrary set of binary classifiers. Based upon rank correlations, our method facilitates a visual comparison of classifiers with arbitrary scores. While the approach is fully versatile and can be applied to any detection task, we demonstrate the method using scores produced by automatic speaker verification and voice anti-spoofing systems.
  arXiv  Detail & Related papers  (2021-06-11T13:03:33Z)
• Automatic Difficulty Classification of Arabic Sentences [0.0]
  The accuracy of our 3-way CEFR classification is F-1 of 0.80 and 0.75 for Arabic-Bert and XLM-R classification respectively and 0.71 Spearman correlation for regression. We compare the use of sentence embeddings of different kinds (fastText, mBERT, XLM-R and Arabic-BERT) as well as traditional language features such as POS tags, dependency trees, readability scores and frequency lists for language learners.
  arXiv  Detail & Related papers  (2021-03-07T16:02:04Z)
• CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognition [14.815422751109061]
  We propose a continuous integrate-and-fire (CIF) based model that supports contextual biasing in a more controllable fashion. An extra context processing network is introduced to extract contextual embeddings, integrate acoustically relevant context information and decode the contextual output distribution. Our method brings relative character error rate (CER) reduction of 8.83%/21.13% and relative named entity character error rate (NE-CER) reduction of 40.14%/51.50% when compared with a strong baseline.
  arXiv  Detail & Related papers  (2020-12-17T09:40:11Z)
• Generating Fact Checking Explanations [52.879658637466605]
  A crucial piece of the puzzle that is still missing is to understand how to automate the most elaborate part of the process. This paper provides the first study of how these explanations can be generated automatically based on available claim context. Our results indicate that optimising both objectives at the same time, rather than training them separately, improves the performance of a fact checking system.
  arXiv  Detail & Related papers  (2020-04-13T05:23:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.