When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence

• URL: http://arxiv.org/abs/2606.26473v1
• Date: Thu, 25 Jun 2026 00:14:22 GMT
• Title: When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence
• Abstract summary: We propose a simple diagnostic to test whether reliability information is used during inference.<n>Experiments on stress recognition and CMU-MOSEI for sentiment analysis show that permuting reliability scores leaves performance unchanged.
• Score: 3.899171236602393
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many multimodal systems estimate the reliability of each modality and weight their contributions to the final prediction. However, it remains unclear whether these scores influence model decisions or merely correlate with performance. We propose a simple diagnostic to test whether reliability information is used during inference. After training, the model and inputs are fixed while reliability scores are permuted across test examples. If predictions depend on these scores, performance should degrade. Experiments on StressID for stress recognition and CMU-MOSEI for sentiment analysis show that permuting reliability scores leaves performance unchanged despite substantial potential gains from selecting the best modality per example. In positive controls where reliability signals identify the correct modality, the same frozen fusion rules yield significant improvements, indicating that reliability signals influence fused decisions only when they reliably predict unimodal correctness.

Related papers

• When Confidence Signals Disagree: Local and Global Confidence in Autoregressive Language Models [0.0]
  We investigate whether different confidence readouts are empirically interchangeable in an autoregressive language model.<n>Across MMLU and ARC Challenge, the two signals are weakly correlated and differ substantially in their association with correctness.<n>Larger local--global confidence gaps are associated with higher answer entropy, more distinct sampled answers, and lower modal-answer concentration.
  arXiv  Detail & Related papers  (2026-09-15T10:08:48Z)
• Post-Calibration Reliability Reranking of Relevance Decisions via Label-wise Monotone Projection [35.67041829703622]
  Web search, product search, and question-answering retrieval systems often assign a relevance label and confidence score to each query-candidate pair.<n>Post-hoc calibration is needed because misaligned confidence can make systems over-trust wrong predictions or unnecessarily defer correct ones.<n>We address this gap with Label-wise Monotone Reliability Projection (MRP), which learns label-wise monotone functions that map calibrated confidence to correctness reliability.
  arXiv  Detail & Related papers  (2026-08-11T02:51:35Z)
• Rethinking Modality Reliability in Multimodal Sentiment Analysis with Incomplete Observations [9.993152436695421]
  We argue that modality reliability is a central variable in incomplete-observation settings. Failure to model it explicitly gives rise to two related issues.<n>We propose Modality Reliability-Calibrated Framework for Multimodal Sentiment Analysis with incomplete observations.<n> MRCF contains a Reliability-Aware Branch that estimates sample-specific modality reliability from intramodal quality cues and cross-modal semantic consistency.<n>Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS show that MRCF achieves strong performance under standard incomplete-observation protocols.
  arXiv  Detail & Related papers  (2026-08-04T13:02:48Z)
• Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations [55.76952683833966]
  This paper proposes weighted CONCH (W-CONCH) and weighted CROC (W-CROC) to reduce confidence set size when data may be corrupted.<n>Experiments on image-based and real-world changepoint and root-cause benchmarks show that uncertainty-based weighting substantially reduces confidence set size.
  arXiv  Detail & Related papers  (2026-07-29T05:16:59Z)
• Mitigating Visual Hallucinations in Multimodal Systems through Retrieval-Augmented Reliability-Aware Inference [1.064167691614925]
  This work proposes a retrieval-augmented reliability-aware inference framework for trustworthy multimodal visual understanding.<n>The proposed framework constructs an external visual evidence database using pretrained visual embeddings and nearest-neighbor retrieval over normalized feature representations.<n>Experiments on ImageNet-100 demonstrate that the proposed reliability-aware framework improves accepted prediction accuracy from 85.84% to 88.88% at 89.04% coverage.
  arXiv  Detail & Related papers  (2026-06-14T12:32:38Z)
• Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning [63.24624171679711]
  Large language model (LLM)-based agents often make suboptimal tool-use decisions.<n>We propose TRUST, which incorporates uncertainty quantification into reward design as a repulsive force for maintaining uncertainty separation.
  arXiv  Detail & Related papers  (2026-06-05T07:08:34Z)
• LLMs Show No Signs Of Individuated Metacognition [0.023227405857540805]
  We decompose binary confidence judgements from 20 Large Language Models.<n>We ask whether two models that differ in confidence also differ in performance.<n>We find no evidence for significant verbalised individuated metacognition in any tested domain.
  arXiv  Detail & Related papers  (2026-05-22T23:54:33Z)
• Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know" [47.930782177987446]
  Large language models often struggle to recognize their knowledge limits in closed-book question answering, leading to confident hallucinations.<n>We evaluate three task-equivalent prompting regimes: Direct, Assistive, and Incremental, across different model scales and multi-hop QA benchmarks.<n>Because factual knowledge is stable while hallucinations are agreement, cross-regime provides a precise signal of internal uncertainty.
  arXiv  Detail & Related papers  (2026-02-04T18:39:58Z)
• Decision-Aware Trust Signal Alignment for SOC Alert Triage [0.0]
  The present paper presents a decision-sensitive trust signal correspondence scheme of SOC alert triage.<n>The framework combines confidence that has been calibrated, lightweight uncertainty cues, and cost-sensitive decision thresholds into coherent decision-support layer.<n>We show that false negatives are greatly amplified by the presence of misaligned displays of confidence, whereas cost weighted loss decreases by orders of magnitude between models with decision aligned trust signals.
  arXiv  Detail & Related papers  (2026-01-08T01:41:54Z)
• Proximity-Based Evidence Retrieval for Uncertainty-Aware Neural Networks [6.9681910774977815]
  This work proposes an evidence-retrieval mechanism for uncertainty-aware decision-making.<n>For each test instance, exemplars are retrieved in an embedding space; their predictive distributions are fused via Dempster-Shafer theory.<n>Because the supporting evidences are explicit, decisions are transparent and auditable.
  arXiv  Detail & Related papers  (2025-09-11T13:12:22Z)
• Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
  Conformal prediction provides model-agnostic and distribution-free uncertainty quantification.<n>Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data.<n>We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
  arXiv  Detail & Related papers  (2024-10-13T15:37:11Z)
• Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [65.21599711087538]
  Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample.<n>Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications.<n>We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
  arXiv  Detail & Related papers  (2024-03-18T05:49:45Z)
• Reliability-Aware Prediction via Uncertainty Learning for Person Image Retrieval [51.83967175585896]
  UAL aims at providing reliability-aware predictions by considering data uncertainty and model uncertainty simultaneously. Data uncertainty captures the noise" inherent in the sample, while model uncertainty depicts the model's confidence in the sample's prediction.
  arXiv  Detail & Related papers  (2022-10-24T17:53:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.