Large Language Models are Few-Shot Training Example Generators: A Case
Study in Fallacy Recognition
- URL: http://arxiv.org/abs/2311.09552v1
- Date: Thu, 16 Nov 2023 04:17:47 GMT
- Title: Large Language Models are Few-Shot Training Example Generators: A Case
Study in Fallacy Recognition
- Authors: Tariq Alhindi, Smaranda Muresan and Preslav Nakov
- Abstract summary: computational fallacy recognition faces challenges due to diverse genres, domains, and types of fallacies found in datasets.
We aim to enhance existing models for fallacy recognition by incorporating additional context and by leveraging large language models to generate synthetic data.
Our evaluation results demonstrate consistent improvements across fallacy types, datasets, and generators.
- Score: 53.952381499149965
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recognizing fallacies is crucial for ensuring the quality and validity of
arguments across various domains. However, computational fallacy recognition
faces challenges due to the diverse genres, domains, and types of fallacies
found in datasets. This leads to a highly multiclass, and even multi-label,
setup with substantial class imbalance. In this study, we aim to enhance
existing models for fallacy recognition by incorporating additional context and
by leveraging large language models to generate synthetic data, thus increasing
the representation of the infrequent classes. We experiment with GPT3.5 to
generate synthetic examples and we examine the impact of prompt settings for
this. Moreover, we explore zero-shot and few-shot scenarios to evaluate the
effectiveness of using the generated examples for training smaller models
within a unified fallacy recognition framework. Furthermore, we analyze the
overlap between the synthetic data and existing fallacy datasets. Finally, we
investigate the usefulness of providing supplementary context for detecting
fallacy types that need such context, e.g., diversion fallacies. Our evaluation
results demonstrate consistent improvements across fallacy types, datasets, and
generators.
Related papers
- Missci: Reconstructing Fallacies in Misrepresented Science [84.32990746227385]
Health-related misinformation on social networks can lead to poor decision-making and real-world dangers.
Missci is a novel argumentation theoretical model for fallacious reasoning.
We present Missci as a dataset to test the critical reasoning abilities of large language models.
arXiv Detail & Related papers (2024-06-05T12:11:10Z) - Tackling Diverse Minorities in Imbalanced Classification [80.78227787608714]
Imbalanced datasets are commonly observed in various real-world applications, presenting significant challenges in training classifiers.
We propose generating synthetic samples iteratively by mixing data samples from both minority and majority classes.
We demonstrate the effectiveness of our proposed framework through extensive experiments conducted on seven publicly available benchmark datasets.
arXiv Detail & Related papers (2023-08-28T18:48:34Z) - Case-Based Reasoning with Language Models for Classification of Logical
Fallacies [3.511369967593153]
We propose a Case-Based Reasoning method that classifies new cases of logical fallacy.
Our experiments indicate that Case-Based Reasoning improves the accuracy and generalizability of language models.
arXiv Detail & Related papers (2023-01-27T17:49:16Z) - Multitask Instruction-based Prompting for Fallacy Recognition [35.10919984256853]
We show the ability of a multitask prompting approach to recognize 28 unique fallacies across domains and genres.
We also study the effect of model size and prompt choice by analyzing the per-class (i.e., fallacy type) results.
arXiv Detail & Related papers (2023-01-24T13:39:23Z) - Towards Generating Adversarial Examples on Mixed-type Data [32.41305735919529]
We propose a novel attack algorithm M-Attack, which can effectively generate adversarial examples in mixed-type data.
Based on M-Attack, attackers can attempt to mislead the targeted classification model's prediction, by only slightly perturbing both the numerical and categorical features in the given data samples.
Our generated adversarial examples can evade potential detection models, which makes the attack indeed insidious.
arXiv Detail & Related papers (2022-10-17T20:17:21Z) - Generalization Properties of Retrieval-based Models [50.35325326050263]
Retrieval-based machine learning methods have enjoyed success on a wide range of problems.
Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored.
We present a formal treatment of retrieval-based models to characterize their generalization ability.
arXiv Detail & Related papers (2022-10-06T00:33:01Z) - On the Transferability of Adversarial Attacksagainst Neural Text
Classifier [121.6758865857686]
We investigate the transferability of adversarial examples for text classification models.
We propose a genetic algorithm to find an ensemble of models that can induce adversarial examples to fool almost all existing models.
We derive word replacement rules that can be used for model diagnostics from these adversarial examples.
arXiv Detail & Related papers (2020-11-17T10:45:05Z) - PermuteAttack: Counterfactual Explanation of Machine Learning Credit
Scorecards [0.0]
This paper is a note on new directions and methodologies for validation and explanation of Machine Learning (ML) models employed for retail credit scoring in finance.
Our proposed framework draws motivation from the field of Artificial Intelligence (AI) security and adversarial ML.
arXiv Detail & Related papers (2020-08-24T00:05:13Z) - Plausible Counterfactuals: Auditing Deep Learning Classifiers with
Realistic Adversarial Examples [84.8370546614042]
Black-box nature of Deep Learning models has posed unanswered questions about what they learn from data.
Generative Adversarial Network (GAN) and multi-objectives are used to furnish a plausible attack to the audited model.
Its utility is showcased within a human face classification task, unveiling the enormous potential of the proposed framework.
arXiv Detail & Related papers (2020-03-25T11:08:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.