Multi-annotator Deep Learning: A Probabilistic Framework for
Classification
- URL: http://arxiv.org/abs/2304.02539v2
- Date: Mon, 23 Oct 2023 23:08:02 GMT
- Title: Multi-annotator Deep Learning: A Probabilistic Framework for
Classification
- Authors: Marek Herde, Denis Huseljic, Bernhard Sick
- Abstract summary: Training standard deep neural networks leads to subpar performances in multi-annotator supervised learning settings.
We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL)
A modular network architecture enables us to make varying assumptions regarding annotators' performances.
Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.
- Score: 2.445702550853822
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Solving complex classification tasks using deep neural networks typically
requires large amounts of annotated data. However, corresponding class labels
are noisy when provided by error-prone annotators, e.g., crowdworkers. Training
standard deep neural networks leads to subpar performances in such
multi-annotator supervised learning settings. We address this issue by
presenting a probabilistic training framework named multi-annotator deep
learning (MaDL). A downstream ground truth and an annotator performance model
are jointly trained in an end-to-end learning approach. The ground truth model
learns to predict instances' true class labels, while the annotator performance
model infers probabilistic estimates of annotators' performances. A modular
network architecture enables us to make varying assumptions regarding
annotators' performances, e.g., an optional class or instance dependency.
Further, we learn annotator embeddings to estimate annotators' densities within
a latent space as proxies of their potentially correlated annotations. Together
with a weighted loss function, we improve the learning from correlated
annotation patterns. In a comprehensive evaluation, we examine three research
questions about multi-annotator supervised learning. Our findings show MaDL's
state-of-the-art performance and robustness against many correlated, spamming
annotators.
Related papers
- Toward Multi-class Anomaly Detection: Exploring Class-aware Unified Model against Inter-class Interference [67.36605226797887]
We introduce a Multi-class Implicit Neural representation Transformer for unified Anomaly Detection (MINT-AD)
By learning the multi-class distributions, the model generates class-aware query embeddings for the transformer decoder.
MINT-AD can project category and position information into a feature embedding space, further supervised by classification and prior probability loss functions.
arXiv Detail & Related papers (2024-03-21T08:08:31Z) - Complementary Learning Subnetworks for Parameter-Efficient
Class-Incremental Learning [40.13416912075668]
We propose a rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks.
Our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order.
arXiv Detail & Related papers (2023-06-21T01:43:25Z) - An Additive Instance-Wise Approach to Multi-class Model Interpretation [53.87578024052922]
Interpretable machine learning offers insights into what factors drive a certain prediction of a black-box system.
Existing methods mainly focus on selecting explanatory input features, which follow either locally additive or instance-wise approaches.
This work exploits the strengths of both methods and proposes a global framework for learning local explanations simultaneously for multiple target classes.
arXiv Detail & Related papers (2022-07-07T06:50:27Z) - CHALLENGER: Training with Attribution Maps [63.736435657236505]
We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance.
In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
arXiv Detail & Related papers (2022-05-30T13:34:46Z) - BatchFormer: Learning to Explore Sample Relationships for Robust
Representation Learning [93.38239238988719]
We propose to enable deep neural networks with the ability to learn the sample relationships from each mini-batch.
BatchFormer is applied into the batch dimension of each mini-batch to implicitly explore sample relationships during training.
We perform extensive experiments on over ten datasets and the proposed method achieves significant improvements on different data scarcity applications.
arXiv Detail & Related papers (2022-03-03T05:31:33Z) - Learning Debiased and Disentangled Representations for Semantic
Segmentation [52.35766945827972]
We propose a model-agnostic and training scheme for semantic segmentation.
By randomly eliminating certain class information in each training iteration, we effectively reduce feature dependencies among classes.
Models trained with our approach demonstrate strong results on multiple semantic segmentation benchmarks.
arXiv Detail & Related papers (2021-10-31T16:15:09Z) - pRSL: Interpretable Multi-label Stacking by Learning Probabilistic Rules [0.0]
We present the probabilistic rule stacking (pRSL) which uses probabilistic propositional logic rules and belief propagation to combine the predictions of several underlying classifiers.
We derive algorithms for exact and approximate inference and learning, and show that pRSL reaches state-of-the-art performance on various benchmark datasets.
arXiv Detail & Related papers (2021-05-28T14:06:21Z) - Adaptive Prototypical Networks with Label Words and Joint Representation
Learning for Few-Shot Relation Classification [17.237331828747006]
This work focuses on few-shot relation classification (FSRC)
We propose an adaptive mixture mechanism to add label words to the representation of the class prototype.
Experiments have been conducted on FewRel under different few-shot (FS) settings.
arXiv Detail & Related papers (2021-01-10T11:25:42Z) - Learning What Makes a Difference from Counterfactual Examples and
Gradient Supervision [57.14468881854616]
We propose an auxiliary training objective that improves the generalization capabilities of neural networks.
We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task.
Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
arXiv Detail & Related papers (2020-04-20T02:47:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.