Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers

• URL: http://arxiv.org/abs/2204.01334v1
• Date: Mon, 4 Apr 2022 09:07:54 GMT
• Title: Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers
• Authors: Jakob Smedegaard Andersen, Walid Maalej
• Abstract summary: We propose a framework for the efficient, in-operation moderation of classifiers' output. We suggest a semi-automated approach that uses prediction uncertainties to pass unconfident, probably incorrect classifications to human moderators. A series of benchmarking experiments show that our framework can improve the classification F1-scores by 5.1 to 11.2%.
• Score: 8.883733362171034
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To maximize the accuracy and increase the overall acceptance of text classifiers, we propose a framework for the efficient, in-operation moderation of classifiers' output. Our framework focuses on use cases in which F1-scores of modern Neural Networks classifiers (ca.~90%) are still inapplicable in practice. We suggest a semi-automated approach that uses prediction uncertainties to pass unconfident, probably incorrect classifications to human moderators. To minimize the workload, we limit the human moderated data to the point where the accuracy gains saturate and further human effort does not lead to substantial improvements. A series of benchmarking experiments based on three different datasets and three state-of-the-art classifiers show that our framework can improve the classification F1-scores by 5.1 to 11.2% (up to approx.~98 to 99%), while reducing the moderation load up to 73.3% compared to a random moderation.

Related papers

• Predicting Overtakes in Trucks Using CAN Data [51.28632782308621]
  We investigate the detection of truck overtakes from CAN data. Our analysis covers up to 10 seconds before the overtaking event. We observe that the prediction scores of the overtake class tend to increase as we approach the overtake trigger.
  arXiv  Detail & Related papers  (2024-04-08T17:58:22Z)
• Class-Imbalanced Semi-Supervised Learning for Large-Scale Point Cloud Semantic Segmentation via Decoupling Optimization [64.36097398869774]
  Semi-supervised learning (SSL) has been an active research topic for large-scale 3D scene understanding. The existing SSL-based methods suffer from severe training bias due to class imbalance and long-tail distributions of the point cloud data. We introduce a new decoupling optimization framework, which disentangles feature representation learning and classifier in an alternative optimization manner to shift the bias decision boundary effectively.
  arXiv  Detail & Related papers  (2024-01-13T04:16:40Z)
• How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks [1.4502611532302039]
  We show that a simple $p$-norm normalization of the logits, followed by taking the maximum logit as the confidence estimator, can lead to considerable gains in selective classification performance. Our results are shown to be consistent under distribution shift.
  arXiv  Detail & Related papers  (2023-05-24T18:56:55Z)
• Robustifying Sentiment Classification by Maximally Exploiting Few Counterfactuals [16.731183915325584]
  We propose a novel solution that only requires annotation of a small fraction of the original training data. We achieve noticeable accuracy improvements by adding only 1% manual counterfactuals.
  arXiv  Detail & Related papers  (2022-10-21T08:30:09Z)
• (Certified!!) Adversarial Robustness for Free! [116.6052628829344]
  We certify 71% accuracy on ImageNet under adversarial perturbations constrained to be within a 2-norm of 0.5. We obtain these results using only pretrained diffusion models and image classifiers, without requiring any fine tuning or retraining of model parameters.
  arXiv  Detail & Related papers  (2022-06-21T17:27:27Z)
• Selective Classification Via Neural Network Training Dynamics [26.58209894993386]
  We show that state-of-the-art selective classification performance can be attained solely from studying the training dynamics of a model. Our method achieves state-of-the-art accuracy/coverage trade-offs on typical selective classification benchmarks.
  arXiv  Detail & Related papers  (2022-05-26T17:51:29Z)
• Interpolation-based Contrastive Learning for Few-Label Semi-Supervised Learning [43.51182049644767]
  Semi-supervised learning (SSL) has long been proved to be an effective technique to construct powerful models with limited labels. Regularization-based methods which force the perturbed samples to have similar predictions with the original ones have attracted much attention. We propose a novel contrastive loss to guide the embedding of the learned network to change linearly between samples.
  arXiv  Detail & Related papers  (2022-02-24T06:00:05Z)
• Semantic Perturbations with Normalizing Flows for Improved Generalization [62.998818375912506]
  We show that perturbations in the latent space can be used to define fully unsupervised data augmentations. We find that our latent adversarial perturbations adaptive to the classifier throughout its training are most effective.
  arXiv  Detail & Related papers  (2021-08-18T03:20:00Z)
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data [78.69828864672978]
  A central challenge in training classification models in the real-world federated system is learning with non-IID data. We propose a novel and simple algorithm called Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated ssian mixture model. Experimental results demonstrate that CCVR state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10.
  arXiv  Detail & Related papers  (2021-06-09T12:02:29Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.