Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking

• URL: http://arxiv.org/abs/2603.19744v1
• Date: Fri, 20 Mar 2026 08:29:05 GMT
• Title: Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking
• Abstract summary: Human Label Variation (HLV) remains underexplored in benchmarks despite rapid progress in large language model development.<n>We introduce an evaluation protocol for multimodal large language model benchmarking that explicitly accounts for two conditions: (1) human label agreement and (2) disagreement.
• Score: 0.03219669032304847
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human Label Variation (HLV), i.e. systematic differences among annotators' judgments, remains underexplored in benchmarks despite rapid progress in large language model (LLM) development. We address this gap by introducing an evaluation protocol for multimodal large language model (MLLM) benchmarking that explicitly accounts for two conditions: (1) human label agreement and (2) disagreement. We apply this protocol to two state-of-the-art MLLM families (Gemma 3, Qwen 2.5 VL) using non-aggregated human annotations from a social media content classification dataset. Across tasks, we find that larger models tend to perform best on high-agreement subsets, yet often underperform medium-sized models when human disagreement is high, indicating that parameter count alone does not determine sensitivity to ambiguity and subjectivity. These results show that benchmarks based solely on consensus labels can overstate model capabilities in such domains and that incorporating human label variation yields more realistic and robust assessments of MLLMs in content moderation pipelines.

Related papers

• Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study [13.666045109234107]
  We analyze BERTScore and SemDist across different Large Language Models (LLMs) for Automatic Speech Recognition (ASR) evaluation.<n>Our results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.
  arXiv  Detail & Related papers  (2026-08-26T09:36:48Z)
• Evaluation of Automatic Speech Recognition Using Generative Large Language Models [13.666045109234107]
  This paper evaluates relevance of Large Language Models (LLMs) for automatic speech recognition evaluation.<n>The best LLMs achieve 92--94% agreement with human annotators for hypothesis selection.<n> Embeddings from decoder-based LLMs show performance comparable to encoder models.
  arXiv  Detail & Related papers  (2026-04-23T17:59:47Z)
• UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning [101.62386137855704]
  We present a novel Universal Multimodal Embedding (UniME-V2) model.<n>Our approach first constructs a potential hard negative set through global retrieval.<n>We then introduce the MLLM-as-a-Judge mechanism, which utilizes MLLMs to assess the semantic alignment of query-candidate pairs.<n>These scores serve as a foundation for hard negative mining, mitigating the impact of false negatives and enabling the identification of diverse, high-quality hard negatives.
  arXiv  Detail & Related papers  (2025-10-15T13:07:00Z)
• Uncovering the Computational Ingredients of Human-Like Representations in LLMs [8.00888290370075]
  It remains unclear which of these ingredients are most crucial for building models that develop human-like representations.<n>Most current benchmarks are not suited to measuring representational alignment between humans and models.
  arXiv  Detail & Related papers  (2025-10-01T15:37:19Z)
• Mapping Overlaps in Benchmarks through Perplexity in the Wild [8.321258152814986]
  We develop signatures of capacity familiarity to characterize large language model (LLM) benchmarks and their meaningful overlaps.<n>Our analysis situates signatures in relation to both the semantic similarity of benchmark questions and the correlation of model performance.<n>Ultimately, we identify cross-functional overlaps across logic, math, language, instruction following, and world modeling, with coding emerging as the least overlapping domain.
  arXiv  Detail & Related papers  (2025-09-27T20:23:13Z)
• Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric [99.56567010306807]
  Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications.<n>One core challenge of evaluation in the large language model (LLM) era is the generalization issue.<n>We propose Model Utilization Index (MUI), a mechanism interpretability enhanced metric that complements traditional performance scores.
  arXiv  Detail & Related papers  (2025-04-10T04:09:47Z)
• Estimating Commonsense Plausibility through Semantic Shifts [66.06254418551737]
  We propose ComPaSS, a novel discriminative framework that quantifies commonsense plausibility by measuring semantic shifts.<n> Evaluations on two types of fine-grained commonsense plausibility estimation tasks show that ComPaSS consistently outperforms baselines.
  arXiv  Detail & Related papers  (2025-02-19T06:31:06Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Open-Domain Text Evaluation via Contrastive Distribution Methods [75.59039812868681]
  We introduce a novel method for evaluating open-domain text generation called Contrastive Distribution Methods. Our experiments on coherence evaluation for multi-turn dialogue and commonsense evaluation for controllable generation demonstrate CDM's superior correlate with human judgment.
  arXiv  Detail & Related papers  (2023-06-20T20:37:54Z)
• A comprehensive comparative evaluation and analysis of Distributional Semantic Models [61.41800660636555]
  We perform a comprehensive evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. The results show that the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous. We borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models.
  arXiv  Detail & Related papers  (2021-05-20T15:18:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.