Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention

• URL: http://arxiv.org/abs/2603.27187v1
• Date: Sat, 28 Mar 2026 08:29:15 GMT
• Title: Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention
• Abstract summary: We introduce OMD-Bench, where all modalities are initially congruent.<n>We then systematically corrupt to isolate each modality's contribution.<n>The benchmark comprises 4,080 instances spanning 27 anchors across eight corruption conditions.
• Score: 7.218987720005679
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing omni-modal benchmarks attempt to measure modality-specific contributions, but their measurements are confounded: naturally co-occurring modalities carry correlated yet unequal information, making it unclear whether results reflect true modality reliance or information asymmetry. We introduce OMD-Bench, where all modalities are initially congruent - each presenting the same anchor, an object or event independently perceivable through video, audio, and text - which we then systematically corrupt to isolate each modality's contribution. We also evaluate calibrated abstention: whether models appropriately refrain from answering when evidence is conflicting. The benchmark comprises 4,080 instances spanning 27 anchors across eight corruption conditions. Evaluating ten omni-modal models under zero-shot and chain-of-thought prompting, we find that models over-abstain when two modalities are corrupted yet under-abstain severely when all three are, while maintaining high confidence (~60-100%) even under full corruption. Chain-of-thought prompting improves abstention alignment with human judgment but amplifies overconfidence rather than mitigating it. OMD-Bench provides a diagnostic benchmark for diagnosing modality reliance, robustness to cross-modal inconsistency, and uncertainty calibration in omni-modal systems.

Related papers

• Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts [49.97842835825998]
  Existing benchmarks conflate two forms of evidence within a single modality: perceptual signals and propositional signals.<n>We introduce Tri-PvP, a benchmark crossing vision, audio, and text, where vision and audio each take perceptual or propositional form.<n>We find robust visual bias across most models and evidence-type conditions. Crucially, we reveal a systematic asymmetry in evidence-form bias.
  arXiv  Detail & Related papers  (2026-09-05T10:33:58Z)
• Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning [5.307937307387054]
  We introduce SCEval, a diagnostic evaluation protocol that keeps the question, answer space, and modality channels fixed while applying controlled structural corruptions to text, vision, and audio individually and jointly.<n>Results show that structural corruption lowers clean accuracy, text--vision damage forms the most stable shared fault line, and multi-modal degradation is non-additive rather than a simple function of the number of corrupted modalities.
  arXiv  Detail & Related papers  (2026-08-29T14:05:42Z)
• Conformal Fusion Under Missing Modalities [6.599344783327054]
  We argue that missing-modality robustness and calibrated uncertainty are a single coupled property, and introduce Modality-Conditioned Conformal Fusion (MCCF)<n>MCCF combines a multimodal bottleneck fusion backbone trained with modality dropout, per-modality evidential heads producing modality-decomposed Dirichlet distributions, and a Dempster-Shafer combination rule.<n>A Mondrian conformal calibration module keyed on the modality-presence mask then provides finite-sample group-conditional coverage for every non-empty modality subset.
  arXiv  Detail & Related papers  (2026-08-07T12:57:16Z)
• Adaptive Modality Reliability Diagnosis and Restoration for Robust Multimodal Intent Recognition [2.2012643583422347]
  Multimodal intent recognition combines linguistic, acoustic, and visual evidence, but individual modalities may be noisy, missing, semantically conflicting, or disproportionately dominant.<n>We propose PRIME, a closed-loop reliability guided framework that jointly diagnoses, restores, and reassesses modality quality at the sample level.<n> Experiments on multimodal intent-recognition benchmarks show that PRIME maintains competitive clean-data performance while improving under missing, noisy, conflicting, and modality-imbalanced conditions.
  arXiv  Detail & Related papers  (2026-08-04T11:14:13Z)
• LLMs Show No Signs Of Individuated Metacognition [0.023227405857540805]
  We decompose binary confidence judgements from 20 Large Language Models.<n>We ask whether two models that differ in confidence also differ in performance.<n>We find no evidence for significant verbalised individuated metacognition in any tested domain.
  arXiv  Detail & Related papers  (2026-05-22T23:54:33Z)
• Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities [20.8430910234166]
  We propose a two-level reference alignment framework to improve robustness under modality missing.<n>First-level reference alignment leverages complete-modality samples to constrain representations and align different modality combinations into a shared sentiment space.<n>Second-level reference alignment enforces cross-modal consistency at the decision level by suppressing unreliable modalities through prototype retrieval and voting.
  arXiv  Detail & Related papers  (2026-05-16T09:03:31Z)
• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems [52.83669998269706]
  Abstention has been studied in text-only settings but remains underexplored multimodally.<n>Current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes.<n>We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes.
  arXiv  Detail & Related papers  (2026-04-16T09:23:22Z)
• Unified Multimodal Uncertain Inference [81.64340342055327]
  Unified Multimodal Uncertain Inference (UMUI) is a task spanning text, audio, and video.<n>We introduce CLUE (Calibrated Latent Uncertainty Estimation), which combines self-consistent teacher calibration and distribution-based calibrated confidence probing to produce predictions.
  arXiv  Detail & Related papers  (2026-04-09T18:46:34Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• Adaptive Multimodal Person Recognition: A Robust Framework for Handling Missing Modalities [2.5472580243871623]
  We propose a Trimodal person identification framework that integrates voice, face, and gesture modalities.<n>Our approach leverages multi-task learning to process each modality independently, followed by a cross-attention and gated fusion mechanism.<n>We show that our system maintains high accuracy even when one or two modalities are unavailable.
  arXiv  Detail & Related papers  (2025-12-16T22:59:24Z)
• Benchmarking Corruption Robustness of LVLMs: A Discriminative Benchmark and Robustness Alignment Metric [49.393713730706445]
  We introduce Bench-C, a benchmark emphasizing discriminative samples for assessing corruption robustness.<n>We propose the Robustness Alignment Score (RAS), a unified metric that measures degradation in logit-level prediction structure.
  arXiv  Detail & Related papers  (2025-11-24T12:07:56Z)
• XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models [29.42489557439947]
  We introduce XModBench, a large-scale tri-modal benchmark to measure cross-modal consistency.<n>XModBench comprises 60,828 multiple-choice questions spanning five task families.<n>Experiments show that even the strongest model, Gemini 2.5 Pro, struggles with spatial and temporal reasoning.
  arXiv  Detail & Related papers  (2025-10-16T21:10:22Z)
• SVeritas: Benchmark for Robust Speaker Verification under Diverse Conditions [54.34001921326444]
  Speaker verification (SV) models are increasingly integrated into security, personalization, and access control systems.<n>Existing benchmarks evaluate only subsets of these conditions, missing others entirely.<n>We introduce SVeritas, a comprehensive Speaker Verification tasks benchmark suite, assessing SV systems under stressors like recording duration, spontaneity, content, noise, microphone distance, reverberation, channel mismatches, audio bandwidth, codecs, speaker age, and susceptibility to spoofing and adversarial attacks.
  arXiv  Detail & Related papers  (2025-09-21T14:11:16Z)
• Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustness [61.87055159919641]
  Multi-modal semantic segmentation (MMSS) addresses the limitations of single-modality data by integrating complementary information across modalities.<n>Despite notable progress, a significant gap persists between research and real-world deployment due to variability and uncertainty in multi-modal data quality.<n>We introduce a robustness benchmark that evaluates MMSS models under three scenarios: Entire-Missing Modality (EMM), Random-Missing Modality (RMM), and Noisy Modality (NM)
  arXiv  Detail & Related papers  (2025-03-24T08:46:52Z)
• A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition [53.800937914403654]
  Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. We propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality.
  arXiv  Detail & Related papers  (2024-03-07T06:06:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.