Multi-View Knowledge Distillation from Crowd Annotations for Out-of-Domain Generalization

• URL: http://arxiv.org/abs/2212.09409v2
• Date: Tue, 23 May 2023 14:44:24 GMT
• Title: Multi-View Knowledge Distillation from Crowd Annotations for Out-of-Domain Generalization
• Authors: Dustin Wright and Isabelle Augenstein
• Abstract summary: We propose new methods for acquiring soft-labels from crowd-annotations by aggregating the distributions produced by existing methods. We demonstrate that these aggregation methods lead to the most consistent performance across four NLP tasks on out-of-domain test sets.
• Score: 53.24606510691877
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Selecting an effective training signal for tasks in natural language processing is difficult: expert annotations are expensive, and crowd-sourced annotations may not be reliable. At the same time, recent work in NLP has demonstrated that learning from a distribution over labels acquired from crowd annotations can be effective. However, there are many ways to acquire such a distribution, and the performance allotted by any one method can fluctuate based on the task and the amount of available crowd annotations, making it difficult to know a priori which distribution is best. This paper systematically analyzes this in the out-of-domain setting, adding to the NLP literature which has focused on in-domain evaluation, and proposes new methods for acquiring soft-labels from crowd-annotations by aggregating the distributions produced by existing methods. In particular, we propose to aggregate multiple-views of crowd annotations via temperature scaling and finding their Jensen-Shannon centroid. We demonstrate that these aggregation methods lead to the most consistent performance across four NLP tasks on out-of-domain test sets, mitigating fluctuations in performance from the individual distributions. Additionally, aggregation results in the most consistently well-calibrated uncertainty estimation. We argue that aggregating different views of crowd-annotations is an effective and minimal intervention to acquire soft-labels which induce robust classifiers despite the inconsistency of the individual soft-labeling methods.

Related papers

• Self-Knowledge Distillation for Learning Ambiguity [11.755814660833549]
  Recent language models often over-confidently predict a single label without consideration for its correctness. We propose a novel self-knowledge distillation method that enables models to learn label distributions more accurately. We validate our method on diverse NLU benchmark datasets and the experimental results demonstrate its effectiveness in producing better label distributions.
  arXiv  Detail & Related papers  (2024-06-14T05:11:32Z)
• Robust Zero-Shot Crowd Counting and Localization With Adaptive Resolution SAM [55.93697196726016]
  We propose a simple yet effective crowd counting method by utilizing the Segment-Everything-Everywhere Model (SEEM) We show that SEEM's performance in dense crowd scenes is limited, primarily due to the omission of many persons in high-density areas. Our proposed method achieves the best unsupervised performance in crowd counting, while also being comparable to some supervised methods.
  arXiv  Detail & Related papers  (2024-02-27T13:55:17Z)
• Class-Distribution-Aware Pseudo Labeling for Semi-Supervised Multi-Label Learning [97.88458953075205]
  Pseudo-labeling has emerged as a popular and effective approach for utilizing unlabeled data. This paper proposes a novel solution called Class-Aware Pseudo-Labeling (CAP) that performs pseudo-labeling in a class-aware manner.
  arXiv  Detail & Related papers  (2023-05-04T12:52:18Z)
• Crowd Counting with Sparse Annotation [28.793141115957564]
  We argue that sparse labeling can reduce redundancy of full annotation and capture more diverse information from distant individuals. We propose a point-based Progressive Point Matching network (PPM) to explore the crowd from the whole image with sparse annotation. Our experimental results show that PPM outperforms previous semi-supervised crowd counting methods with the same amount of annotation by a large margin.
  arXiv  Detail & Related papers  (2023-04-12T17:57:48Z)
• Neighbour Consistency Guided Pseudo-Label Refinement for Unsupervised Person Re-Identification [80.98291772215154]
  Unsupervised person re-identification (ReID) aims at learning discriminative identity features for person retrieval without any annotations. Recent advances accomplish this task by leveraging clustering-based pseudo labels. We propose a Neighbour Consistency guided Pseudo Label Refinement framework.
  arXiv  Detail & Related papers  (2022-11-30T09:39:57Z)
• Rethinking Clustering-Based Pseudo-Labeling for Unsupervised Meta-Learning [146.11600461034746]
  Method for unsupervised meta-learning, CACTUs, is a clustering-based approach with pseudo-labeling. This approach is model-agnostic and can be combined with supervised algorithms to learn from unlabeled data. We prove that the core reason for this is lack of a clustering-friendly property in the embedding space.
  arXiv  Detail & Related papers  (2022-09-27T19:04:36Z)
• An Additive Instance-Wise Approach to Multi-class Model Interpretation [53.87578024052922]
  Interpretable machine learning offers insights into what factors drive a certain prediction of a black-box system. Existing methods mainly focus on selecting explanatory input features, which follow either locally additive or instance-wise approaches. This work exploits the strengths of both methods and proposes a global framework for learning local explanations simultaneously for multiple target classes.
  arXiv  Detail & Related papers  (2022-07-07T06:50:27Z)
• Feature Diversity Learning with Sample Dropout for Unsupervised Domain Adaptive Person Re-identification [0.0]
  This paper proposes a new approach to learn the feature representation with better generalization ability through limiting noisy pseudo labels. We put forward a brand-new method referred as to Feature Diversity Learning (FDL) under the classic mutual-teaching architecture. Experimental results show that our proposed FDL-SD achieves the state-of-the-art performance on multiple benchmark datasets.
  arXiv  Detail & Related papers  (2022-01-25T10:10:48Z)
• Learning from Crowds with Sparse and Imbalanced Annotations [29.596070201105274]
  crowdsourcing has established itself as an efficient labeling solution through resorting to non-expert crowds. One common practice is to distribute each instance to multiple workers, whereas each worker only annotates a subset of data, resulting in the it sparse annotation phenomenon. We propose one self-training based approach named it Self-Crowd by progressively adding confident pseudo-annotations and rebalancing the annotation distribution.
  arXiv  Detail & Related papers  (2021-07-11T13:06:20Z)
• CrowdTeacher: Robust Co-teaching with Noisy Answers & Sample-specific Perturbations for Tabular Data [8.276156981100364]
  Co-teaching methods have shown promising improvements for computer vision problems with noisy labels. Our model, CrowdTeacher, uses the idea that robustness in the input space model can improve the perturbation of the classifier for noisy labels. We showcase the boost in predictive power attained using CrowdTeacher for both synthetic and real datasets.
  arXiv  Detail & Related papers  (2021-03-31T15:09:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.