Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments

• URL: http://arxiv.org/abs/2212.07425v3
• Date: Mon, 25 Sep 2023 04:47:16 GMT
• Title: Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments
• Authors: Zhivar Sourati, Vishnu Priya Prasanna Venkatesh, Darshan Deshpande, Himanshu Rawlani, Filip Ilievski, H\^ong-\^An Sandlin, Alain Mermoud
• Abstract summary: We formalize prior theoretical work on logical fallacies into a comprehensive three-stage evaluation framework. We employ three families of robust and explainable methods based on prototype reasoning, instance-based reasoning, and knowledge injection. We extensively evaluate these methods on our datasets, focusing on their robustness and explainability.
• Score: 5.850977561881791
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The spread of misinformation, propaganda, and flawed argumentation has been amplified in the Internet era. Given the volume of data and the subtlety of identifying violations of argumentation norms, supporting information analytics tasks, like content moderation, with trustworthy methods that can identify logical fallacies is essential. In this paper, we formalize prior theoretical work on logical fallacies into a comprehensive three-stage evaluation framework of detection, coarse-grained, and fine-grained classification. We adapt existing evaluation datasets for each stage of the evaluation. We employ three families of robust and explainable methods based on prototype reasoning, instance-based reasoning, and knowledge injection. The methods combine language models with background knowledge and explainable mechanisms. Moreover, we address data sparsity with strategies for data augmentation and curriculum learning. Our three-stage framework natively consolidates prior datasets and methods from existing tasks, like propaganda detection, serving as an overarching evaluation testbed. We extensively evaluate these methods on our datasets, focusing on their robustness and explainability. Our results provide insight into the strengths and weaknesses of the methods on different components and fallacy classes, indicating that fallacy identification is a challenging task that may require specialized forms of reasoning to capture various classes. We share our open-source code and data on GitHub to support further work on logical fallacy identification.

Related papers

• EventGround: Narrative Reasoning by Grounding to Eventuality-centric Knowledge Graphs [41.928535719157054]
  We propose an initial comprehensive framework called EventGround to tackle the problem of grounding free-texts to eventuality-centric knowledge graphs. We provide simple yet effective parsing and partial information extraction methods to tackle these problems. Our framework, incorporating grounded knowledge, achieves state-of-the-art performance while providing interpretable evidence.
  arXiv  Detail & Related papers  (2024-03-30T01:16:37Z)
• Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data [89.2410799619405]
  We introduce the Quantitative Reasoning with Data benchmark to evaluate Large Language Models' capability in statistical and causal reasoning with real-world data. The benchmark comprises a dataset of 411 questions accompanied by data sheets from textbooks, online learning materials, and academic papers. To compare models' quantitative reasoning abilities on data and text, we enrich the benchmark with an auxiliary set of 290 text-only questions, namely QRText.
  arXiv  Detail & Related papers  (2024-02-27T16:15:03Z)
• Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition [49.38757847011105]
  computational fallacy recognition faces challenges due to diverse genres, domains, and types of fallacies found in datasets. We aim to enhance existing models for fallacy recognition by incorporating additional context and by leveraging large language models to generate synthetic data. Our evaluation results demonstrate consistent improvements across fallacy types, datasets, and generators.
  arXiv  Detail & Related papers  (2023-11-16T04:17:47Z)
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
  We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
  arXiv  Detail & Related papers  (2023-11-14T07:13:10Z)
• A Study of Situational Reasoning for Traffic Understanding [63.45021731775964]
  We devise three novel text-based tasks for situational reasoning in the traffic domain. We adopt four knowledge-enhanced methods that have shown generalization capability across language reasoning tasks in prior work. We provide in-depth analyses of model performance on data partitions and examine model predictions categorically.
  arXiv  Detail & Related papers  (2023-06-05T01:01:12Z)
• Multitask Instruction-based Prompting for Fallacy Recognition [35.10919984256853]
  We show the ability of a multitask prompting approach to recognize 28 unique fallacies across domains and genres. We also study the effect of model size and prompt choice by analyzing the per-class (i.e., fallacy type) results.
  arXiv  Detail & Related papers  (2023-01-24T13:39:23Z)
• Distant finetuning with discourse relations for stance classification [55.131676584455306]
  We propose a new method to extract data with silver labels from raw text to finetune a model for stance classification. We also propose a 3-stage training framework where the noisy level in the data used for finetuning decreases over different stages. Our approach ranks 1st among 26 competing teams in the stance classification track of the NLPCC 2021 shared task Argumentative Text Understanding for AI Debater.
  arXiv  Detail & Related papers  (2022-04-27T04:24:35Z)
• MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning [63.50909998372667]
  We propose MERIt, a MEta-path guided contrastive learning method for logical ReasonIng of text. Two novel strategies serve as indispensable components of our method.
  arXiv  Detail & Related papers  (2022-03-01T11:13:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.