QuMAB: Query-based Multi-Annotator Behavior Modeling with Reliability under Sparse Labels

• URL: http://arxiv.org/abs/2507.17653v2
• Date: Thu, 07 Aug 2025 15:17:29 GMT
• Title: QuMAB: Query-based Multi-Annotator Behavior Modeling with Reliability under Sparse Labels
• Abstract summary: Multi-annotator learning traditionally aggregates diverse annotations to approximate a single ground truth, treating disagreements as noise.<n>We introduce a paradigm shift from sample-wise aggregation to annotator-wise behavior modeling.<n>By treating annotator disagreements as valuable information rather than noise, modeling annotator-specific behavior patterns can reconstruct unlabeled data to reduce annotation cost, enhance aggregation reliability, and explain annotator decision behavior.
• Score: 23.555446749682467
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-annotator learning traditionally aggregates diverse annotations to approximate a single ground truth, treating disagreements as noise. However, this paradigm faces fundamental challenges: subjective tasks often lack absolute ground truth, and sparse annotation coverage makes aggregation statistically unreliable. We introduce a paradigm shift from sample-wise aggregation to annotator-wise behavior modeling. By treating annotator disagreements as valuable information rather than noise, modeling annotator-specific behavior patterns can reconstruct unlabeled data to reduce annotation cost, enhance aggregation reliability, and explain annotator decision behavior. To this end, we propose QuMAB (Query-based Multi-Annotator Behavior Pattern Learning), which uses light-weight queries to model individual annotators while capturing inter-annotator correlations as implicit regularization, preventing overfitting to sparse individual data while maintaining individualization and improving generalization, with a visualization of annotator focus regions offering an explainable analysis of behavior understanding. We contribute two large-scale datasets with dense per-annotator labels: STREET (4,300 labels/annotator) and AMER (average 3,118 labels/annotator), the first multimodal multi-annotator dataset. Extensive experiments demonstrate the superiority of our QuMAB in modeling individual annotators' behavior patterns, their utility for consensus prediction, and applicability under sparse annotations.

Related papers

• The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction [39.10305114526259]
  We introduce a framework combining conformal prediction with Collaborative Filtering-style annotators' representation.<n>Using Non-Conformity Scores, we introduce the Ghost Prediction metric and the Ghost Annotator representation to quantify cases in which model predictions diverge from all available human annotations.<n>Our finding shows that while all models uncertainty increases with annotator disagreement, larger models tend to be more confident in the classification of texts that are not aligned with any human annotation.
  arXiv  Detail & Related papers  (2026-06-01T21:32:37Z)
• Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization [51.93456979139756]
  Free-text explanations extend human label variation beyond label disagreement.<n>We study whether large language models can learn and reproduce such annotator-specific label-explanation behavior.
  arXiv  Detail & Related papers  (2026-05-27T17:55:00Z)
• REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations [13.49289449502791]
  We propose REALM, a method that jointly learns the model parameters and a scalar expertise value for each annotator entirely unsupervised.<n>We evaluate on five question answering benchmarks, fine-tuning three sizes of Flan-T5 under simulated noisy annotations.<n>The proposed algorithm consistently outperforms the naive noisy SFT in the large majority of single- and multi-task settings.
  arXiv  Detail & Related papers  (2026-04-19T07:06:35Z)
• Merge and Guide: Unifying Model Merging and Guided Decoding for Controllable Multi-Objective Generation [49.98025799046136]
  We introduce Merge-And-GuidE, a two-stage framework that leverages model merging for guided decoding.<n>In Stage 1, MAGE resolves a compatibility problem between the guidance and base models.<n>In Stage 2, we merge explicit and implicit value models into a unified guidance proxy, which then steers the decoding of the base model from Stage 1.
  arXiv  Detail & Related papers  (2025-10-04T11:10:07Z)
• Stochastic Encodings for Active Feature Acquisition [100.47043816019888]
  Active Feature Acquisition is an instance-wise, sequential decision making problem.<n>The aim is to dynamically select which feature to measure based on current observations, independently for each test instance.<n>Common approaches either use Reinforcement Learning, which experiences training difficulties, or greedily maximize the conditional mutual information of the label and unobserved features, which makes myopic.<n>We introduce a latent variable model, trained in a supervised manner. Acquisitions are made by reasoning about the features across many possible unobserved realizations in a latent space.
  arXiv  Detail & Related papers  (2025-08-03T23:48:46Z)
• BBQRec: Behavior-Bind Quantization for Multi-Modal Sequential Recommendation [15.818669767036592]
  We propose a Behavior-Bind multi-modal Quantization for Sequential Recommendation (BBQRec) featuring dual-aligned quantization and semantics-aware sequence modeling.<n>BBQRec disentangles modality-agnostic behavioral patterns from noisy modality-specific features through contrastive codebook learning.<n>We design a discretized similarity reweighting mechanism that dynamically adjusts self-attention scores using quantized semantic relationships.
  arXiv  Detail & Related papers  (2025-04-09T07:19:48Z)
• A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks [51.14185612418977]
  A strategy to improve label quality is to ask multiple annotators to label the same item and aggregate their labels. While a variety of bespoke models have been proposed for specific tasks, our work is the first to introduce aggregation methods that generalize across many diverse complex tasks. This article extends our prior work with investigation of three new research questions.
  arXiv  Detail & Related papers  (2023-12-20T21:28:35Z)
• Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles [104.60508550106618]
  We propose DiffDiv, an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs)<n>We show that DPMs can generate images with novel feature combinations, even when trained on samples displaying correlated input features.<n>We show that DPM-guided diversification is sufficient to remove dependence on shortcut cues, without a need for additional supervised signals.
  arXiv  Detail & Related papers  (2023-11-23T15:47:33Z)
• ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation [35.10805667891489]
  Active learning, as an annotation cost-saving strategy, has not been fully explored in the context of learning from disagreement. We show that in the active learning setting, a multi-head model performs significantly better than a single-head model in terms of uncertainty estimation.
  arXiv  Detail & Related papers  (2023-10-23T14:26:43Z)
• Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks [92.32670915472099]
  We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs) We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
  arXiv  Detail & Related papers  (2023-10-03T17:37:52Z)
• Transferring Annotator- and Instance-dependent Transition Matrix for Learning from Crowds [88.06545572893455]
  In real-world crowd-sourcing scenarios, noise transition matrices are both annotator- and instance-dependent. We first model the mixture of noise patterns by all annotators, and then transfer this modeling to individual annotators. Experiments confirm the superiority of the proposed approach on synthetic and real-world crowd-sourcing data.
  arXiv  Detail & Related papers  (2023-06-05T13:43:29Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Utilizing supervised models to infer consensus labels and their quality from data with multiple annotators [16.79939549201032]
  Real-world data for classification is often labeled by multiple annotators. We introduce CROWDLAB, a straightforward approach to estimate such data. Our proposed method provides superior estimates for (1)- (3) than many alternative algorithms.
  arXiv  Detail & Related papers  (2022-10-13T07:54:07Z)
• Disjoint Contrastive Regression Learning for Multi-Sourced Annotations [10.159313152511919]
  Large-scale datasets are important for the development of deep learning models. Multiple annotators may be employed to label different subsets of the data. The inconsistency and bias among different annotators are harmful to the model training.
  arXiv  Detail & Related papers  (2021-12-31T12:39:04Z)
• Joint Multi-Dimensional Model for Global and Time-Series Annotations [48.159050222769494]
  Crowdsourcing is a popular approach to collect annotations for unlabeled data instances. It involves collecting a large number of annotations from several, often naive untrained annotators for each data instance which are then combined to estimate the ground truth. Most annotation fusion schemes however ignore this aspect and model each dimension separately. We propose a generative model for multi-dimensional annotation fusion, which models the dimensions jointly leading to more accurate ground truth estimates.
  arXiv  Detail & Related papers  (2020-05-06T20:08:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.