MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation

• URL: http://arxiv.org/abs/2605.06080v1
• Date: Thu, 07 May 2026 12:02:23 GMT
• Title: MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation
• Abstract summary: We propose MSD-Score, a reference-free metric that models image patch and text token embeddings as von Mises-Fisher mixtures on the unit hypersphere.<n>Semantic discrepancies are quantified via a weighted bi-directional KL divergence and combined with global similarity.<n>Experiments show that MSD-Score achieves state-of-the-art correlation with human judgments among reference-free metrics.
• Score: 32.34482125102006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating image captions without references remains challenging because global embedding similarity often misses fine-grained mismatches such as hallucinated objects, missing attributes, or incorrect relations. We propose MSD-Score, a reference-free metric that models image patch and text token embeddings as von Mises-Fisher mixtures on the unit hypersphere. Instead of treating each modality as a single point, MSD-Score formulates image-text matching as a multi-scale distributional scoring problem. Semantic discrepancies are quantified via a weighted bi-directional KL divergence and combined with global similarity in a multi-scale framework for both single- and multi-candidate evaluations. Extensive experiments show that MSD-Score achieves state-of-the-art correlation with human judgments among reference-free metrics. Beyond accuracy, its probabilistic formulation yields transparent and decomposable diagnostics of local grounding errors, providing a deterministic complementary signal to holistic similarity metrics and judge-based evaluators.

Related papers

• Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment [48.882448108261826]
  Infrared-Visible image fusion (IVIF) aims to integrate thermal information and detailed spatial structures into a single fused image to enhance perception.<n>Recent IVIF reward-modelling efforts learn from human ratings but use scalar regression on aggregated scores.<n>We introduce FuScore, which utilizes an MLLM to mimic human visual perception by producing continuous quality score.
  arXiv  Detail & Related papers  (2026-05-07T21:38:09Z)
• Federated Measurement of Demographic Disparities from Quantile Sketches [4.991212094743681]
  Horizontal federated learning (FL) enables collaborative modeling across clients with aligned features without sharing raw data.<n>We study federated auditing of demographic parity through score distributions, measuring disparity as a Wasserstein--Frechet variance between sensitive-group score laws.<n>We propose a one-shot, communication-efficient protocol in which each silo shares only group counts and a quantile summary of its local score distributions.
  arXiv  Detail & Related papers  (2026-02-21T15:34:44Z)
• Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics [25.374192139098284]
  We study prototypicality bias as a systematic failure mode in multimodal evaluation.<n>We introduce a controlled contrastive benchmark ProtoBias, spanning Animals, Objects, and Demography images.<n>Our results show that widely used metrics, including CLIPScore, PickScore, and VQA-based scores, frequently misrank these pairs.<n>We propose ProtoScore, a robust 7B- parameter metric that substantially reduces failure rates and suppresses misranking.
  arXiv  Detail & Related papers  (2026-01-08T13:49:14Z)
• MS-ISSM: Objective Quality Assessment of Point Clouds Using Multi-scale Implicit Structural Similarity [65.85858856481131]
  unstructured and irregular nature of point clouds poses a significant challenge for objective quality assessment (PCQA)<n>We propose the Multi-scale Implicit Structural Similarity Measurement (MS-ISSM)
  arXiv  Detail & Related papers  (2026-01-03T14:58:52Z)
• The Impact of the Single-Label Assumption in Image Recognition Benchmarking [1.4828022319975973]
  Deep neural networks (DNNs) are typically evaluated under the assumption that each image has a single correct label.<n>Many images in benchmarks like ImageNet contain multiple valid labels, creating a mismatch between evaluation protocols and the actual complexity of visual data.<n>We rigorously assess the impact of multi-label characteristics on reported accuracy gaps.
  arXiv  Detail & Related papers  (2024-12-24T12:55:31Z)
• HICEScore: A Hierarchical Metric for Image Captioning Evaluation [10.88292081473071]
  We propose a novel reference-free metric for image captioning evaluation, dubbed Hierarchical Image Captioning Evaluation Score (HICE-S) By detecting local visual regions and textual phrases, HICE-S builds an interpretable hierarchical scoring mechanism. Our proposed metric achieves the SOTA performance on several benchmarks, outperforming existing reference-free metrics.
  arXiv  Detail & Related papers  (2024-07-26T08:24:30Z)
• Dynamic Correlation Learning and Regularization for Multi-Label Confidence Calibration [60.95748658638956]
  This paper introduces the Multi-Label Confidence task, aiming to provide well-calibrated confidence scores in multi-label scenarios. Existing single-label calibration methods fail to account for category correlations, which are crucial for addressing semantic confusion. We propose the Dynamic Correlation Learning and Regularization algorithm, which leverages multi-grained semantic correlations to better model semantic confusion.
  arXiv  Detail & Related papers  (2024-07-09T13:26:21Z)
• Rank-Consistency Deep Hashing for Scalable Multi-Label Image Search [90.30623718137244]
  We propose a novel deep hashing method for scalable multi-label image search. A new rank-consistency objective is applied to align the similarity orders from two spaces. A powerful loss function is designed to penalize the samples whose semantic similarity and hamming distance are mismatched.
  arXiv  Detail & Related papers  (2021-02-02T13:46:58Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)
• Out-of-distribution Generalization via Partial Feature Decorrelation [72.96261704851683]
  We present a novel Partial Feature Decorrelation Learning (PFDL) algorithm, which jointly optimize a feature decomposition network and the target image classification model. The experiments on real-world datasets demonstrate that our method can improve the backbone model's accuracy on OOD image classification datasets.
  arXiv  Detail & Related papers  (2020-07-30T05:48:48Z)
• Joint Visual and Temporal Consistency for Unsupervised Domain Adaptive Person Re-Identification [64.37745443119942]
  This paper jointly enforces visual and temporal consistency in the combination of a local one-hot classification and a global multi-class classification. Experimental results on three large-scale ReID datasets demonstrate the superiority of proposed method in both unsupervised and unsupervised domain adaptive ReID tasks.
  arXiv  Detail & Related papers  (2020-07-21T14:31:27Z)
• Reliable Fidelity and Diversity Metrics for Generative Models [30.941563781926202]
  The most widely used metric for measuring the similarity between real and generated images has been the Fr'echet Inception Distance (FID) score. We show that even the latest version of the precision and recall metrics are not reliable yet. We propose density and coverage metrics that solve the above issues.
  arXiv  Detail & Related papers  (2020-02-23T00:50:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.