Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition

• URL: http://arxiv.org/abs/2311.09552v2
• Date: Thu, 15 Aug 2024 15:00:12 GMT
• Title: Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition
• Authors: Tariq Alhindi, Smaranda Muresan, Preslav Nakov,
• Abstract summary: computational fallacy recognition faces challenges due to diverse genres, domains, and types of fallacies found in datasets. We aim to enhance existing models for fallacy recognition by incorporating additional context and by leveraging large language models to generate synthetic data. Our evaluation results demonstrate consistent improvements across fallacy types, datasets, and generators.
• Score: 49.38757847011105
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recognizing fallacies is crucial for ensuring the quality and validity of arguments across various domains. However, computational fallacy recognition faces challenges due to the diverse genres, domains, and types of fallacies found in datasets. This leads to a highly multi-class, and even multi-label, setup with substantial class imbalance. In this study, we aim to enhance existing models for fallacy recognition by incorporating additional context and by leveraging large language models to generate synthetic data, thus increasing the representation of the infrequent classes. We experiment with GPT3.5 to generate synthetic examples and we examine the impact of prompt settings for this. Moreover, we explore zero-shot and few-shot scenarios to evaluate the effectiveness of using the generated examples for training smaller models within a unified fallacy recognition framework. Furthermore, we analyze the overlap between the synthetic data and existing fallacy datasets. Finally, we investigate the usefulness of providing supplementary context for detecting fallacy types that need such context, e.g., diversion fallacies. Our evaluation results demonstrate consistent improvements across fallacy types, datasets, and generators. The code and the synthetic datasets are all publicly available.

Related papers

• A Logical Fallacy-Informed Framework for Argument Generation [34.35377699079075]
  We introduce FIPO, a fallacy-informed framework that steers Large Language Models toward logically sound arguments. Our results on argumentation datasets show that our method reduces the fallacy errors by up to 17.5%. Our code is available atlucamouchel.com/lucamouchel/Logical-Fallacies.
  arXiv  Detail & Related papers  (2024-08-07T08:19:44Z)
• Synthetic Oversampling: Theory and A Practical Approach Using LLMs to Address Data Imbalance [16.047084318753377]
  Imbalanced data and spurious correlations are common challenges in machine learning and data science. Oversampling, which artificially increases the number of instances in the underrepresented classes, has been widely adopted to tackle these challenges. We introduce OPAL, a systematic oversampling approach that leverages the capabilities of large language models to generate high-quality synthetic data for minority groups.
  arXiv  Detail & Related papers  (2024-06-05T21:24:26Z)
• Missci: Reconstructing Fallacies in Misrepresented Science [84.32990746227385]
  Health-related misinformation on social networks can lead to poor decision-making and real-world dangers. Missci is a novel argumentation theoretical model for fallacious reasoning. We present Missci as a dataset to test the critical reasoning abilities of large language models.
  arXiv  Detail & Related papers  (2024-06-05T12:11:10Z)
• Tackling Diverse Minorities in Imbalanced Classification [80.78227787608714]
  Imbalanced datasets are commonly observed in various real-world applications, presenting significant challenges in training classifiers. We propose generating synthetic samples iteratively by mixing data samples from both minority and majority classes. We demonstrate the effectiveness of our proposed framework through extensive experiments conducted on seven publicly available benchmark datasets.
  arXiv  Detail & Related papers  (2023-08-28T18:48:34Z)
• Case-Based Reasoning with Language Models for Classification of Logical Fallacies [3.511369967593153]
  We propose a Case-Based Reasoning method that classifies new cases of logical fallacy. Our experiments indicate that Case-Based Reasoning improves the accuracy and generalizability of language models.
  arXiv  Detail & Related papers  (2023-01-27T17:49:16Z)
• Multitask Instruction-based Prompting for Fallacy Recognition [35.10919984256853]
  We show the ability of a multitask prompting approach to recognize 28 unique fallacies across domains and genres. We also study the effect of model size and prompt choice by analyzing the per-class (i.e., fallacy type) results.
  arXiv  Detail & Related papers  (2023-01-24T13:39:23Z)
• Generalization Properties of Retrieval-based Models [50.35325326050263]
  Retrieval-based machine learning methods have enjoyed success on a wide range of problems. Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored. We present a formal treatment of retrieval-based models to characterize their generalization ability.
  arXiv  Detail & Related papers  (2022-10-06T00:33:01Z)
• On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study [65.17429512679695]
  In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions. Despite ADC's intuitive appeal, it remains unclear when training on adversarial datasets produces more robust models.
  arXiv  Detail & Related papers  (2021-06-02T00:48:33Z)
• On the Transferability of Adversarial Attacksagainst Neural Text Classifier [121.6758865857686]
  We investigate the transferability of adversarial examples for text classification models. We propose a genetic algorithm to find an ensemble of models that can induce adversarial examples to fool almost all existing models. We derive word replacement rules that can be used for model diagnostics from these adversarial examples.
  arXiv  Detail & Related papers  (2020-11-17T10:45:05Z)
• PermuteAttack: Counterfactual Explanation of Machine Learning Credit Scorecards [0.0]
  This paper is a note on new directions and methodologies for validation and explanation of Machine Learning (ML) models employed for retail credit scoring in finance. Our proposed framework draws motivation from the field of Artificial Intelligence (AI) security and adversarial ML.
  arXiv  Detail & Related papers  (2020-08-24T00:05:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.