Meta-learning Representations for Learning from Multiple Annotators

• URL: http://arxiv.org/abs/2506.10259v1
• Date: Thu, 12 Jun 2025 00:58:37 GMT
• Title: Meta-learning Representations for Learning from Multiple Annotators
• Authors: Atsutoshi Kumagai, Tomoharu Iwata, Taishi Nishiyama, Yasutoshi Ida, Yasuhiro Fujiwara,
• Abstract summary: We propose a meta-learning method for learning from multiple noisy annotators.<n>The proposed method embeds each example in tasks to a latent space by using a neural network.<n>We show the effectiveness of our method with real-world datasets with synthetic noise and real-world crowdsourcing datasets.
• Score: 40.886894995806955
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a meta-learning method for learning from multiple noisy annotators. In many applications such as crowdsourcing services, labels for supervised learning are given by multiple annotators. Since the annotators have different skills or biases, given labels can be noisy. To learn accurate classifiers, existing methods require many noisy annotated data. However, sufficient data might be unavailable in practice. To overcome the lack of data, the proposed method uses labeled data obtained in different but related tasks. The proposed method embeds each example in tasks to a latent space by using a neural network and constructs a probabilistic model for learning a task-specific classifier while estimating annotators' abilities on the latent space. This neural network is meta-learned to improve the expected test classification performance when the classifier is adapted to a given small amount of annotated data. This classifier adaptation is performed by maximizing the posterior probability via the expectation-maximization (EM) algorithm. Since each step in the EM algorithm is easily computed as a closed-form and is differentiable, the proposed method can efficiently backpropagate the loss through the EM algorithm to meta-learn the neural network. We show the effectiveness of our method with real-world datasets with synthetic noise and real-world crowdsourcing datasets.

Related papers

• Unsupervised Estimation of Ensemble Accuracy [0.0]
  We present a method for estimating the joint power of several classifiers. It differs from existing approaches which focus on "diversity" measures by not relying on labels. We demonstrate the method on popular large-scale face recognition datasets.
  arXiv  Detail & Related papers  (2023-11-18T02:31:36Z)
• Neural Active Learning on Heteroskedastic Distributions [29.01776999862397]
  We demonstrate the catastrophic failure of active learning algorithms on heteroskedastic datasets. We propose a new algorithm that incorporates a model difference scoring function for each data point to filter out the noisy examples and sample clean examples.
  arXiv  Detail & Related papers  (2022-11-02T07:30:19Z)
• Robust Meta-learning with Sampling Noise and Label Noise via Eigen-Reptile [78.1212767880785]
  meta-learner is prone to overfitting since there are only a few available samples. When handling the data with noisy labels, the meta-learner could be extremely sensitive to label noise. We present Eigen-Reptile (ER) that updates the meta- parameters with the main direction of historical task-specific parameters.
  arXiv  Detail & Related papers  (2022-06-04T08:48:02Z)
• Trustable Co-label Learning from Multiple Noisy Annotators [68.59187658490804]
  Supervised deep learning depends on massive accurately annotated examples. A typical alternative is learning from multiple noisy annotators. This paper proposes a data-efficient approach, called emphTrustable Co-label Learning (TCL)
  arXiv  Detail & Related papers  (2022-03-08T16:57:00Z)
• Learning with Neighbor Consistency for Noisy Labels [69.83857578836769]
  We present a method for learning from noisy labels that leverages similarities between training examples in feature space. We evaluate our method on datasets evaluating both synthetic (CIFAR-10, CIFAR-100) and realistic (mini-WebVision, Clothing1M, mini-ImageNet-Red) noise.
  arXiv  Detail & Related papers  (2022-02-04T15:46:27Z)
• BAMLD: Bayesian Active Meta-Learning by Disagreement [39.59987601426039]
  This paper introduces an information-theoretic active task selection mechanism to decrease the number of labeling requests for meta-training tasks. We report its empirical performance results that compare favourably against existing acquisition mechanisms.
  arXiv  Detail & Related papers  (2021-10-19T13:06:51Z)
• The information of attribute uncertainties: what convolutional neural networks can learn about errors in input data [0.0]
  We show how Convolutional Neural Networks (CNNs) are able to learn about the context and patterns of signal and noise. We show that, when each data point is subject to different levels of noise, that information can be learned by the CNNs.
  arXiv  Detail & Related papers  (2021-08-10T15:10:46Z)
• Fast Few-Shot Classification by Few-Iteration Meta-Learning [173.32497326674775]
  We introduce a fast optimization-based meta-learning method for few-shot classification. Our strategy enables important aspects of the base learner objective to be learned during meta-training. We perform a comprehensive experimental analysis, demonstrating the speed and effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-10-01T15:59:31Z)
• An Online Learning Algorithm for a Neuro-Fuzzy Classifier with Mixed-Attribute Data [9.061408029414455]
  General fuzzy min-max neural network (GFMMNN) is one of the efficient neuro-fuzzy systems for data classification. This paper proposes an extended online learning algorithm for the GFMMNN. The proposed method can handle the datasets with both continuous and categorical features.
  arXiv  Detail & Related papers  (2020-09-30T13:45:36Z)
• On the Robustness of Active Learning [0.7340017786387767]
  Active Learning is concerned with how to identify the most useful samples for a Machine Learning algorithm to be trained with. We find that it is often applied with not enough care and domain knowledge. We propose the new "Sum of Squared Logits" method based on the Simpson diversity index and investigate the effect of using the confusion matrix for balancing in sample selection.
  arXiv  Detail & Related papers  (2020-06-18T09:07:23Z)
• Learning with Out-of-Distribution Data for Audio Classification [60.48251022280506]
  We show that detecting and relabelling certain OOD instances, rather than discarding them, can have a positive effect on learning. The proposed method is shown to improve the performance of convolutional neural networks by a significant margin.
  arXiv  Detail & Related papers  (2020-02-11T21:08:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.