HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide

• URL: http://arxiv.org/abs/2608.29193v1
• Date: Sat, 29 Aug 2026 11:06:37 GMT
• Title: HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide
• Abstract summary: We propose HalluPrism, a behavioral diagnostic that re-runs an answer after visual degradation, blank-image replacement, and grounding or relation checks.<n>These targeted probes yield a signature over visual-perturbation sensitivity, image-removal confidence retention, and grounding/relation-probe instability.
• Score: 18.50218795316691
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) can assign similar confidence to answers that fail for different reasons. We propose HalluPrism, a behavioral diagnostic that re-runs an answer after visual degradation, blank-image replacement, and grounding or relation checks. These targeted probes yield a signature over visual-perturbation sensitivity (V ), image-removal confidence retention (L), and grounding/relation-probe instability (A). Across 58K+ examples from four benchmarks and four MLLMs, image-removal confidence retention is most prevalent, while grounding/relation-probe instability better separates failure families. Only 18 of 48 source-target checks are diagonally aligned, so the coordinates should be interpreted jointly rather than as independent causal sources. With the dataset fixed, the joint signature improves failure-family AUROC from 0.634 to 0.769 on HallusionBench and from 0.707 to 0.817 on VizWiz, with smaller gains on POPE and VSR. In pooled XGBoost analysis, AUROC rises from 0.78 with scalar confidence to 0.95 with (V, L, A) and 0.97 when confidence is added. The same signature does not automatically improve correctness ranking. The three tested direct scalarizations can harm it. These results separate failure diagnosis from abstention scoring: multimodal uncertainty should characterize failure structure before it is used to decide whether to abstain or correct.

Related papers

• Not All Relations Are Equal: Relation-Balanced and Calibrated Graph Learning for Provenance-Based Intrusion Detection [66.44527094471619]
  We present RECAL, an unsupervised framework using relation-balanced masked graph learning to better capture rare interaction patterns.<n>On three DARPA E3 datasets, RECAL achieves F1 scores of 99.99%, 99.93%, and 99.99%, outperforming the best baseline on each dataset by 0.88, 0.82, and 0.42 percentage points, respectively.
  arXiv  Detail & Related papers  (2026-09-15T00:31:01Z)
• From Causal Plausibility to Causal Reliability: Evaluating LLMs as Calibrated Direct Causal-Edge Classifiers [15.44881677475537]
  Large language models (LLMs) are increasingly used to provide prior causal knowledge for structural causal discovery.<n>We evaluate 12 instruction-tuned open-weight models across six benchmark causal graphs, five prompting strategies, and four confidence sources.<n>Our results suggest LLMs are better viewed as sources of externally validated soft causal priors than as direct evidence of causal structure.
  arXiv  Detail & Related papers  (2026-08-24T15:55:47Z)
• EVADE: Evidence-Verified Agentic Diagnosis with Escape [0.880899367147235]
  EVADE is an inferential, non-training method that enhances the safety of deploying a single frozen VLM.<n>It commits only when both the entire image and the zoomed view responses agree.<n>It reduces expected calibration error (ECE) by up to 45% compared to zero-shot.
  arXiv  Detail & Related papers  (2026-08-19T11:51:24Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation [0.0]
  We measure, on identical inputs, how model scale and 4-bit quantization affect two confidence signals in the Qwen2-VL family.<n>We find that scale sharply improves the model's internal uncertainty signal.<n>We argue that error-detection AUROC, not calibration error, is the metric that exposes the difference between the two signals.
  arXiv  Detail & Related papers  (2026-07-27T13:47:44Z)
• LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data [2.1443570696048906]
  Large language models (LLMs) are increasingly applied to structured clinical data.<n>We compare Qwen 2.5 7B and XGBoost on a prediction task via attribution divergence analysis.
  arXiv  Detail & Related papers  (2026-06-17T18:49:44Z)
• CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction [51.56484100374058]
  Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness.<n>We introduce textscCaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction.<n>textscCaliDist consistently achieves lower Expected Error (ECE) and Brier Score compared with strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T07:27:53Z)
• How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles [46.63622714488747]
  Shared pretraining data, distillation, and alignment pipelines can induce hidden behavioral dependencies, latent entanglement.<n>In practice, this manifests as correlated reasoning patterns and synchronized failures.<n>We develop a statistical framework for auditing behavioral entanglement among black-box language models.
  arXiv  Detail & Related papers  (2026-04-08T23:32:06Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Impact of Labeling Inaccuracy and Image Noise on Tooth Segmentation in Panoramic Radiographs using Federated, Centralized and Local Learning [46.232038247686745]
  Federated learning (FL) may mitigate privacy constraints, heterogeneous data quality, and inconsistent labeling in dental diagnostic AI.<n>We compared FL with centralized (CL) and local learning (LL) for tooth segmentation in panoramic radiographs across multiple data corruption scenarios.
  arXiv  Detail & Related papers  (2025-09-08T11:07:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.