Generative Score Inference for Multimodal Data

• URL: http://arxiv.org/abs/2603.26349v1
• Date: Fri, 27 Mar 2026 12:24:12 GMT
• Title: Generative Score Inference for Multimodal Data
• Abstract summary: We introduce Generative Score Inference, a flexible inference framework capable of constructing statistically valid and informative prediction and confidence sets.<n>We empirically validate GSI's capabilities through two representative scenarios: hallucination detection in large language models and uncertainty estimation in image captioning.<n>Our method achieves state-of-the-art performance in hallucination detection and robust predictive uncertainty in image captioning, and its performance is positively influenced by the quality of the underlying generative model.
• Score: 11.857867207010981
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate uncertainty quantification is crucial for making reliable decisions in various supervised learning scenarios, particularly when dealing with complex, multimodal data such as images and text. Current approaches often face notable limitations, including rigid assumptions and limited generalizability, constraining their effectiveness across diverse supervised learning tasks. To overcome these limitations, we introduce Generative Score Inference (GSI), a flexible inference framework capable of constructing statistically valid and informative prediction and confidence sets across a wide range of multimodal learning problems. GSI utilizes synthetic samples generated by deep generative models to approximate conditional score distributions, facilitating precise uncertainty quantification without imposing restrictive assumptions about the data or tasks. We empirically validate GSI's capabilities through two representative scenarios: hallucination detection in large language models and uncertainty estimation in image captioning. Our method achieves state-of-the-art performance in hallucination detection and robust predictive uncertainty in image captioning, and its performance is positively influenced by the quality of the underlying generative model. These findings underscore the potential of GSI as a versatile inference framework, significantly enhancing uncertainty quantification and trustworthiness in multimodal learning.

Related papers

• CAMD: Coverage-Aware Multimodal Decoding for Efficient Reasoning of Multimodal Large Language Models [48.33850939857255]
  We develop a theoretical framework that links sampling coverage, instance difficulty, and residual risk.<n>Our analysis reveals that multimodal reasoning exhibits a heavy-tailed difficulty distribution.<n>We propose Coverage-Aware Multimodal Decoding (CAMD), an adaptive inference mechanism that dynamically allocates according to estimated uncertainty.
  arXiv  Detail & Related papers  (2026-03-16T02:31:03Z)
• Heterogeneous Uncertainty-Guided Composed Image Retrieval with Fine-Grained Probabilistic Learning [49.28548464288051]
  Composed Image Retrieval (CIR) enables image search by combining a reference image with modification text.<n>In intrinsic noise in CIR triplets incurs intrinsic uncertainty and threatens the model's robustness.<n>This paper introduces a Heterogeneous Uncertainty-Guided (HUG) paradigm to overcome these limitations.
  arXiv  Detail & Related papers  (2026-01-16T16:05:49Z)
• LLM-Centric RAG with Multi-Granular Indexing and Confidence Constraints [5.2604064919135896]
  This paper addresses the issues of insufficient coverage, unstable results, and limited reliability in retrieval-augmented generation under complex knowledge environments.<n>It proposes a confidence control method that integrates multi-granularity memory indexing with uncertainty estimation.<n>The results show that the method achieves superior performance over existing models in QA accuracy, retrieval recall, ranking quality, and factual consistency.
  arXiv  Detail & Related papers  (2025-10-30T23:48:37Z)
• Reasoning Under Uncertainty: Exploring Probabilistic Reasoning Capabilities of LLMs [47.20307724127832]
  We present the first comprehensive study of the reasoning capabilities of large language models (LLMs)<n>We evaluate models on three carefully designed tasks, mode identification, maximum likelihood estimation, and sample generation.<n>Through empirical evaluations, we demonstrate that there exists a clear performance gap between smaller and larger models.
  arXiv  Detail & Related papers  (2025-09-12T22:58:05Z)
• Enhancing Uncertainty Estimation and Interpretability via Bayesian Non-negative Decision Layer [55.66973223528494]
  We develop a Bayesian Non-negative Decision Layer (BNDL), which reformulates deep neural networks as a conditional Bayesian non-negative factor analysis.<n>BNDL can model complex dependencies and provide robust uncertainty estimation.<n>We also offer theoretical guarantees that BNDL can achieve effective disentangled learning.
  arXiv  Detail & Related papers  (2025-05-28T10:23:34Z)
• A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation [0.0]
  This review consolidates and contextualizes foundational concepts in uncertainty modeling.<n>We identify challenges, such as strong assumptions in spatial aggregation and lack of standardized benchmarks.<n>We propose directions for advancing uncertainty-aware segmentation in deep learning.
  arXiv  Detail & Related papers  (2024-11-25T13:26:09Z)
• The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness [50.52507648690234]
  Federated learning has the risk of skewing fine-tuning features and compromising the robustness of the model. We introduce three robustness indicators and conduct experiments across diverse robust datasets. Our approach markedly enhances the robustness across diverse scenarios, encompassing various parameter-efficient fine-tuning methods.
  arXiv  Detail & Related papers  (2024-01-25T09:18:51Z)
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [61.94125052118442]
  Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL) In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
  arXiv  Detail & Related papers  (2023-03-03T16:12:59Z)
• Toward Robust Uncertainty Estimation with Random Activation Functions [3.0586855806896045]
  We propose a novel approach for uncertainty quantification via ensembles, called Random Activation Functions (RAFs) Ensemble. RAFs Ensemble outperforms state-of-the-art ensemble uncertainty quantification methods on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-02-28T13:17:56Z)
• Modeling Uncertain Feature Representation for Domain Generalization [49.129544670700525]
  We show that our method consistently improves the network generalization ability on multiple vision tasks. Our methods are simple yet effective and can be readily integrated into networks without additional trainable parameters or loss constraints.
  arXiv  Detail & Related papers  (2023-01-16T14:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.