OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lenses

• URL: http://arxiv.org/abs/2608.24160v1
• Date: Tue, 25 Aug 2026 07:18:35 GMT
• Title: OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lenses
• Abstract summary: We introduce D3- Omni, a balanced and decoupled benchmark for diagnosing fine-grained multimodal understanding.<n>Rather than re-generating outputs, which may leak information across dimensions, we fix verified fully positive seeds and derive negatives.<n>The resulting D3 design is Dual-balanced, which helps alleviate negative-sample scarcity and per-dimension label imbalance.
• Score: 15.379006872674106
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multimodal understanding models that can jointly judge text-to-image (T2I), text-to-video (T2V) and text-to-speech (TTS) generation are increasingly used as "OmniJudges" for evaluation and automatic annotation. How reliably they understand what they score remains unclear, since existing benchmarks and training data tend to overemphasize positive examples and to conflate distinct failure modes, so a judge may score well without recognizing failures while its capability gaps stay hidden. Motivated by this, we introduce D3-Omni, a balanced and decoupled benchmark for diagnosing fine-grained multimodal understanding, covering 53 orthogonal binary dimensions (17/22/14) and 10,671 samples (3,526/1,998/5,147) across the three tasks. Rather than re-generating outputs, which may leak information across dimensions, we fix verified fully positive seeds and derive negatives through controlled prompt rewriting and atomic, dimension-isolating perturbations. The resulting D3 design is Dual-balanced, which helps alleviate negative-sample scarcity and per-dimension label imbalance; Decoupled, so that each error is attributable to a single capability; and Dynamic, steering construction toward under-represented regions of the label distribution as generative models improve.The suite reaches near 1:1 per-dimension parity and a uniform distribution over all total-score levels. Under this balanced view, even strong OmniJudges tend to struggle on modality-related dimensions, to confirm satisfied requirements far more reliably than they detect violated ones, and to treat nominally distinct attributes as largely a single decision, suggesting that aggregate accuracy may hide systematic blind spots that a balanced and decoupled lens can help expose and, in turn, address.

Related papers

• When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering [0.8594140167290097]
  A model's agreement across perturbed inputs is used as a label-free reliability signal.<n>Rerendering beats resampling on both accuracy and reliability.<n>Fine-tuning on the model's own cross-render consensus inverts.
  arXiv  Detail & Related papers  (2026-08-06T07:08:05Z)
• Twins: Learn to Predict Unified Representations with Focal Loss [36.63304689132002]
  Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation.<n>We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid.
  arXiv  Detail & Related papers  (2026-07-24T17:59:39Z)
• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• Evian: Towards Explainable Visual Instruction-tuning Data Auditing [14.93566912726999]
  We introduce a novel "Decomposition-then-Evaluation" paradigm that breaks model responses into constituent cognitive components.<n>We instantiate this paradigm via EVIAN, an automated framework that evaluates these components along the axes of Image-Text Consistency, Logical Coherence, and Factual Accuracy.
  arXiv  Detail & Related papers  (2026-04-22T13:28:27Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings [91.3041144166326]
  We propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf vision-language models into robust omni-modal embedding models.<n>e5-omni combines modality-aware temperature calibration to align similarity scales, a controllable negative curriculum with debiasing to focus on confusing negatives, and batch whitening to better match cross-modal geometry.<n> Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines.
  arXiv  Detail & Related papers  (2026-01-07T07:39:40Z)
• Enhancing Dual Network Based Semi-Supervised Medical Image Segmentation with Uncertainty-Guided Pseudo-Labeling [5.1962665598872135]
  This paper proposes a novel semi-supervised 3D medical image segmentation framework based on a dual-network architecture.<n>Specifically, we investigate a Cross Consistency Enhancement module using both cross pseudo and entropy-filtered supervision to reduce the noisy pseudo-labels.<n>In addition, we use a self-supervised contrastive learning mechanism to align uncertain voxel features with reliable class prototypes.
  arXiv  Detail & Related papers  (2025-09-16T13:40:20Z)
• Dual-Stage Reweighted MoE for Long-Tailed Egocentric Mistake Detection [85.0189917888094]
  We propose a Dual-Stage Reweighted Mixture-of-Experts (DR-MoE) framework to handle the challenges posed by subtle and infrequent mistakes.<n>The proposed method achieves strong performance, particularly in identifying rare and ambiguous mistake instances.
  arXiv  Detail & Related papers  (2025-09-16T12:00:42Z)
• Integrating Disparity Confidence Estimation into Relative Depth Prior-Guided Unsupervised Stereo Matching [55.784713740698365]
  Unsupervised stereo matching has garnered significant attention for its independence from costly disparity annotations.<n>A feasible solution lies in transferring 3D geometric knowledge from a relative depth map to the stereo matching networks.<n>This work proposes a novel unsupervised learning framework to address these challenges.
  arXiv  Detail & Related papers  (2025-08-02T09:11:05Z)
• DAC: 2D-3D Retrieval with Noisy Labels via Divide-and-Conquer Alignment and Correction [26.164120380820307]
  We propose a Divide-and-conquer 2D-3D cross-modal Alignment and Correction framework, which comprises Multimodal Dynamic Division (MDD) and Adaptive Alignment and Correction (AAC) In AAC, samples in distinct subsets are exploited with different alignment strategies to fully enhance the semantic compactness and meanwhile over-fitting to noisy labels. To evaluate the effectiveness in real-world scenarios, we introduce a challenging noisy benchmark, namely. N200, which comprises 200k-level samples annotated with 1156 realistic noisy labels.
  arXiv  Detail & Related papers  (2024-07-25T05:18:18Z)
• Uncertainty-Aware Adaptation for Self-Supervised 3D Human Pose Estimation [70.32536356351706]
  We introduce MRP-Net that constitutes a common deep network backbone with two output heads subscribing to two diverse configurations. We derive suitable measures to quantify prediction uncertainty at both pose and joint level. We present a comprehensive evaluation of the proposed approach and demonstrate state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-29T07:14:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.