Flee the Flaw: Annotating the Underlying Logic of Fallacious Arguments Through Templates and Slot-filling

• URL: http://arxiv.org/abs/2406.12402v1
• Date: Tue, 18 Jun 2024 08:44:45 GMT
• Title: Flee the Flaw: Annotating the Underlying Logic of Fallacious Arguments Through Templates and Slot-filling
• Authors: Irfan Robbani, Paul Reisert, Naoya Inoue, Surawat Pothong, Camélia Guerraoui, Wenzhi Wang, Shoichi Naito, Jungmin Choi, Kentaro Inui,
• Abstract summary: We introduce four sets of explainable templates for common informal logical fallacies. We conduct an annotation study on top of 400 fallacious arguments taken from LOGIC dataset. We discover that state-of-the-art language models struggle with detecting fallacy templates.
• Score: 15.339084849719223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prior research in computational argumentation has mainly focused on scoring the quality of arguments, with less attention on explicating logical errors. In this work, we introduce four sets of explainable templates for common informal logical fallacies designed to explicate a fallacy's implicit logic. Using our templates, we conduct an annotation study on top of 400 fallacious arguments taken from LOGIC dataset and achieve a high agreement score (Krippendorf's alpha of 0.54) and reasonable coverage (0.83). Finally, we conduct an experiment for detecting the structure of fallacies and discover that state-of-the-art language models struggle with detecting fallacy templates (0.47 accuracy). To facilitate research on fallacies, we make our dataset and guidelines publicly available.

Related papers

• A Logical Fallacy-Informed Framework for Argument Generation [34.35377699079075]
  We introduce FIPO, a fallacy-informed framework that steers Large Language Models toward logically sound arguments. Our results on argumentation datasets show that our method reduces the fallacy errors by up to 17.5%. Our code is available atlucamouchel.com/lucamouchel/Logical-Fallacies.
  arXiv  Detail & Related papers  (2024-08-07T08:19:44Z)
• Missci: Reconstructing Fallacies in Misrepresented Science [84.32990746227385]
  Health-related misinformation on social networks can lead to poor decision-making and real-world dangers. Missci is a novel argumentation theoretical model for fallacious reasoning. We present Missci as a dataset to test the critical reasoning abilities of large language models.
  arXiv  Detail & Related papers  (2024-06-05T12:11:10Z)
• NL2FOL: Translating Natural Language to First-Order Logic for Logical Fallacy Detection [45.28949266878263]
  We design a process to reliably detect logical fallacies by translating natural language to First-order Logic. We then utilize Satisfiability Modulo Theory (SMT) solvers to reason about the validity of the formula. Our approach is robust, interpretable and does not require training data or fine-tuning.
  arXiv  Detail & Related papers  (2024-04-18T00:20:48Z)
• CASA: Causality-driven Argument Sufficiency Assessment [79.13496878681309]
  We propose CASA, a zero-shot causality-driven argument sufficiency assessment framework. PS measures how likely introducing the premise event would lead to the conclusion when both the premise and conclusion events are absent. Experiments on two logical fallacy detection datasets demonstrate that CASA accurately identifies insufficient arguments.
  arXiv  Detail & Related papers  (2024-01-10T16:21:18Z)
• LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models [63.14196038655506]
  We introduce LogicAsker, a novel approach for evaluating and enhancing the logical reasoning capabilities of large language models (LLMs) Our methodology reveals significant gaps in LLMs' learning of logical rules, with identified reasoning failures ranging from 29% to 90% across different models. We leverage these findings to construct targeted demonstration examples and fine-tune data, notably enhancing logical reasoning in models like GPT-4o by up to 5%.
  arXiv  Detail & Related papers  (2024-01-01T13:53:53Z)
• Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition [49.38757847011105]
  computational fallacy recognition faces challenges due to diverse genres, domains, and types of fallacies found in datasets. We aim to enhance existing models for fallacy recognition by incorporating additional context and by leveraging large language models to generate synthetic data. Our evaluation results demonstrate consistent improvements across fallacy types, datasets, and generators.
  arXiv  Detail & Related papers  (2023-11-16T04:17:47Z)
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
  We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
  arXiv  Detail & Related papers  (2023-11-14T07:13:10Z)
• How susceptible are LLMs to Logical Fallacies? [5.723715910568911]
  We present LOGICOM, a diagnostic benchmark to assess the robustness of Large Language Models against logical fallacies. We use this benchmark to evaluate the performance of GPT-3.5 and GPT-4 using a dataset containing controversial topics. Our findings indicate that both GPT-3.5 and GPT-4 can adjust their opinion through reasoning.
  arXiv  Detail & Related papers  (2023-08-18T23:07:29Z)
• Theme Aspect Argumentation Model for Handling Fallacies [2.3230307339947274]
  We present a novel approach to characterising fallacies through formal constraints. By identifying fallacies with formal constraints, it becomes possible to tell whether a fallacy lurks in the modelled rhetoric with a formal rigour.
  arXiv  Detail & Related papers  (2022-05-30T14:34:09Z)
• Logical Fallacy Detection [40.06349885733248]
  We propose the task of logical fallacy detection, and provide a new dataset (Logic) of logical fallacies generally found in text. We show that a simple structure-aware classifier outperforms the best language model by 5.46% on Logic and 4.51% on LogicClimate.
  arXiv  Detail & Related papers  (2022-02-28T13:18:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.