D-CALM: A Dynamic Clustering-based Active Learning Approach for
Mitigating Bias
- URL: http://arxiv.org/abs/2305.17013v1
- Date: Fri, 26 May 2023 15:17:43 GMT
- Title: D-CALM: A Dynamic Clustering-based Active Learning Approach for
Mitigating Bias
- Authors: Sabit Hassan and Malihe Alikhani
- Abstract summary: In this paper, we propose a novel adaptive clustering-based active learning algorithm, D-CALM, that dynamically adjusts clustering and annotation efforts.
Experiments on eight datasets for a diverse set of text classification tasks, including emotion, hatespeech, dialog act, and book type detection, demonstrate that our proposed algorithm significantly outperforms baseline AL approaches.
- Score: 13.008323851750442
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Despite recent advancements, NLP models continue to be vulnerable to bias.
This bias often originates from the uneven distribution of real-world data and
can propagate through the annotation process. Escalated integration of these
models in our lives calls for methods to mitigate bias without overbearing
annotation costs. While active learning (AL) has shown promise in training
models with a small amount of annotated data, AL's reliance on the model's
behavior for selective sampling can lead to an accumulation of unwanted bias
rather than bias mitigation. However, infusing clustering with AL can overcome
the bias issue of both AL and traditional annotation methods while exploiting
AL's annotation efficiency. In this paper, we propose a novel adaptive
clustering-based active learning algorithm, D-CALM, that dynamically adjusts
clustering and annotation efforts in response to an estimated classifier
error-rate. Experiments on eight datasets for a diverse set of text
classification tasks, including emotion, hatespeech, dialog act, and book type
detection, demonstrate that our proposed algorithm significantly outperforms
baseline AL approaches with both pretrained transformers and traditional
Support Vector Machines. D-CALM showcases robustness against different measures
of information gain and, as evident from our analysis of label and error
distribution, can significantly reduce unwanted model bias.
Related papers
- Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice.
We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
arXiv Detail & Related papers (2023-12-27T09:03:43Z) - XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners [71.8257151788923]
We propose a novel Explainable Active Learning framework (XAL) for low-resource text classification.
XAL encourages classifiers to justify their inferences and delve into unlabeled data for which they cannot provide reasonable explanations.
Experiments on six datasets show that XAL achieves consistent improvement over 9 strong baselines.
arXiv Detail & Related papers (2023-10-09T08:07:04Z) - Taming Small-sample Bias in Low-budget Active Learning [20.900107811622803]
Firth bias reduction can provably reduce the bias during the model training process but might hinder learning if its coefficient is not adaptive to the learning progress.
We propose curriculum Firth bias reduction (CHAIN) that can automatically adjust the coefficient to be adaptive to the training process.
arXiv Detail & Related papers (2023-06-19T16:42:11Z) - Controlling Learned Effects to Reduce Spurious Correlations in Text
Classifiers [6.662800021628275]
We propose an algorithm to regularize the learnt effect of the features on the model's prediction to the estimated effect of feature on label.
On toxicity and IMDB review datasets, the proposed algorithm minimises spurious correlations and improves the minority group.
arXiv Detail & Related papers (2023-05-26T12:15:54Z) - Rethinking Precision of Pseudo Label: Test-Time Adaptation via
Complementary Learning [10.396596055773012]
We propose a novel complementary learning approach to enhance test-time adaptation.
In test-time adaptation tasks, information from the source domain is typically unavailable.
We highlight that the risk function of complementary labels agrees with their Vanilla loss formula.
arXiv Detail & Related papers (2023-01-15T03:36:33Z) - Feature-Level Debiased Natural Language Understanding [86.8751772146264]
Existing natural language understanding (NLU) models often rely on dataset biases to achieve high performance on specific datasets.
We propose debiasing contrastive learning (DCT) to mitigate biased latent features and neglect the dynamic nature of bias.
DCT outperforms state-of-the-art baselines on out-of-distribution datasets while maintaining in-distribution performance.
arXiv Detail & Related papers (2022-12-11T06:16:14Z) - Prototypical Classifier for Robust Class-Imbalanced Learning [64.96088324684683]
We propose textitPrototypical, which does not require fitting additional parameters given the embedding network.
Prototypical produces balanced and comparable predictions for all classes even though the training set is class-imbalanced.
We test our method on CIFAR-10LT, CIFAR-100LT and Webvision datasets, observing that Prototypical obtains substaintial improvements compared with state of the arts.
arXiv Detail & Related papers (2021-10-22T01:55:01Z) - Learning Bias-Invariant Representation by Cross-Sample Mutual
Information Minimization [77.8735802150511]
We propose a cross-sample adversarial debiasing (CSAD) method to remove the bias information misused by the target task.
The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator.
We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.
arXiv Detail & Related papers (2021-08-11T21:17:02Z) - Task-agnostic Continual Learning with Hybrid Probabilistic Models [75.01205414507243]
We propose HCL, a Hybrid generative-discriminative approach to Continual Learning for classification.
The flow is used to learn the data distribution, perform classification, identify task changes, and avoid forgetting.
We demonstrate the strong performance of HCL on a range of continual learning benchmarks such as split-MNIST, split-CIFAR, and SVHN-MNIST.
arXiv Detail & Related papers (2021-06-24T05:19:26Z) - ALEX: Active Learning based Enhancement of a Model's Explainability [34.26945469627691]
An active learning (AL) algorithm seeks to construct an effective classifier with a minimal number of labeled examples in a bootstrapping manner.
In the era of data-driven learning, this is an important research direction to pursue.
This paper describes our work-in-progress towards developing an AL selection function that in addition to model effectiveness also seeks to improve on the interpretability of a model during the bootstrapping steps.
arXiv Detail & Related papers (2020-09-02T07:15:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.