Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity

• URL: http://arxiv.org/abs/2606.20676v1
• Date: Fri, 12 Jun 2026 15:53:40 GMT
• Title: Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
• Abstract summary: We introduce VOIR DIRE, a multimodal benchmark of 626 culturally paired image-prompt artifacts spanning U.S. and mainland Chinese contexts.<n>Across six MLLMs, the bias decomposes into two failures: a positivity-floor calibration failure and an orientation failure.<n>We recommend reporting alignment against each reference pool separately and treating cross-pool divergence as a judge property.
• Score: 43.23157850664282
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: MLLM-as-a-Judge is conventionally validated by agreement with human annotations, but this metric is undefined when the human pool is culturally heterogeneous. We introduce VOIR DIRE, a multimodal benchmark of 626 culturally paired image--prompt artifacts spanning U.S. and mainland Chinese contexts across food, fashion, and architecture, with annotator pools that are within-pool reliable (a = 0.86/0.74) but cross-pool divergent on evaluation (Q1 r = -0.12). Across six MLLMs, the bias decomposes into two failures: a positivity-floor calibration failure (compressed scale use) and an orientation failure (default to one cultural norm). On this corpus, where contested items are sampled to split the two pools, the floor mechanically validates the more-permissive Chinese reading; persona prompting partially recovers calibration, but the orientation residual survives, evidence the tilt is not reducible to scale compression. Reference-pool in-context demonstrations deepen the orientation residual and inflate the high end rather than restoring use of the low end. Model origin adds a small additive tilt (~0.10 MAE) that is approximately invariant under demonstration. We recommend reporting alignment against each reference pool separately and treating cross-pool divergence as a judge property.

Related papers

• Self-Supervised Topologically Invariant Manifold Learning for Railway Image Quality Assessment [20.081901964453074]
  Existing blind image quality assessment (BIQA) methods rely on synthetic subjective annotations, limiting generalization in real-world domains.<n>We propose a fully self-supervised BIQA framework based on topologically invariant manifold learning under boundary constraints.<n>We show that our framework secures superior zero-shot transferability across standard synthetic and wild benchmarks.
  arXiv  Detail & Related papers  (2026-08-15T13:08:06Z)
• MAAM: Anchor-Preserving Compression and Contextual Calibration for Chinese Discriminatory Language Detection [4.939857202189033]
  We propose MAAM (Myopia--Astigmatism Anchor Mechanism), a lightweight, model-agnostic framework inspired by functional visual blur.<n>We also introduce ChLGBT, the first Chinese LGBT-focused discriminatory-language dataset.<n>Across strong encoder baselines, MAAM improves all three prediction dimensions, with consistent gains in accuracy, F1, Brier score, and expected calibration error.
  arXiv  Detail & Related papers  (2026-06-08T07:07:16Z)
• BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence [22.315546054051143]
  Bias audits of large language models now operate within governance frameworks such as the EU AI Act.<n>We introduce textscBiAxisAudit, a protocol that reports each bias score together with a reliability estimate on two axes.
  arXiv  Detail & Related papers  (2026-05-09T16:26:49Z)
• Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIF [0.3227658251731014]
  Large language models (LLMs) frequently exhibit performance biases against regional dialects of low-resource languages.<n>We propose a two-phase framework to evaluate dialectal bias in LLM question-answering across nine Bengali dialects.
  arXiv  Detail & Related papers  (2026-03-22T18:44:57Z)
• Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook [50.10594064510559]
  DOVE is a distributional evaluation framework that directly compares human-written text with LLM-generated outputs.<n>We show that DOVE achieves superior predictive validity, attaining a 31.56% correlation with downstream tasks, while maintaining high reliability with as few as 500 samples per culture.
  arXiv  Detail & Related papers  (2026-03-16T08:33:10Z)
• Are Aligned Large Language Models Still Misaligned? [13.062124372682106]
  Mis-Align Bench is a unified benchmark for analyzing misalignment across safety, value, and cultural dimensions.<n> SAVACU is an English-aligned dataset of 382,424 misaligned samples spanning 112 domains (or labels)
  arXiv  Detail & Related papers  (2026-02-11T19:30:43Z)
• Mitigating Cultural Bias in LLMs via Multi-Agent Cultural Debate [7.517766226036547]
  Large language models (LLMs) exhibit systematic Western-centric bias, yet whether prompting in non-Western languages can mitigate this remains understudied.<n>We introduce CEBiasBench, a Chinese--English bilingual benchmark, and Multi-Agent Vote (MAV), which enables explicit no bias'' judgments.<n>Using this framework, we find that Chinese prompting merely shifts bias toward East Asian perspectives rather than eliminating it.
  arXiv  Detail & Related papers  (2026-01-17T16:00:34Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Better Estimation of the Kullback--Leibler Divergence Between Language Models [62.09293217844222]
  Estimating the Kullback--Leibler (KL) divergence between language models has many applications.<n>We introduce a Rao--Blackwellized estimator that is unbiased and provably has variance less than or equal to that of the standard Monte Carlo estimator.
  arXiv  Detail & Related papers  (2025-04-14T18:40:02Z)
• "Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation [90.09260023184932]
  Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) output by leveraging external knowledge sources to reduce factual hallucinations. NoMIRACL is a human-annotated dataset for evaluating LLM robustness in RAG across 18 typologically diverse languages. We measure relevance assessment using: (i) hallucination rate, measuring model tendency to hallucinate, when the answer is not present in passages in the non-relevant subset, and (ii) error rate, measuring model inaccuracy to recognize relevant passages in the relevant subset.
  arXiv  Detail & Related papers  (2023-12-18T17:18:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.