Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulation

• URL: http://arxiv.org/abs/2505.16180v2
• Date: Wed, 24 Sep 2025 03:01:31 GMT
• Title: Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulation
• Authors: Ashim Dahal, Ankit Ghimire, Saydul Akbar Murad, Nick Rahimi,
• Abstract summary: Redemption Score(RS) is a novel framework that ranks image captions by triangulating three complementary signals.<n>On the Flickr8k benchmark, RS achieves a Kendall-$tau$ of 58.42, outperforming most prior methods.
• Score: 3.4998703934432682
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating image captions requires cohesive assessment of both visual semantics and language pragmatics, which is often not entirely captured by most metrics. We introduce Redemption Score(RS), a novel hybrid framework that ranks image captions by triangulating three complementary signals: (1) Mutual Information Divergence (MID) for global image-text distributional alignment, (2) DINO-based perceptual similarity of cycle-generated images for visual grounding, and (3) LLM Text Embeddings for contextual text similarity against human references. A calibrated fusion of these signals allows RS to offer a more holistic assessment. On the Flickr8k benchmark, RS achieves a Kendall-$\tau$ of 58.42, outperforming most prior methods and demonstrating superior correlation with human judgments without requiring task-specific training. Our framework provides a more robust and nuanced evaluation by thoroughly examining both the visual accuracy and text quality together, with consistent performance across Conceptual Captions and MS COCO.

Related papers

• Cite-While-You-Generate: Training-Free Evidence Attribution for Multimodal Clinical Summarization [32.47484883374212]
  Trustworthy clinical summarization requires fluent generation and transparency about where each statement comes from.<n>We propose a training-free framework for generation-time source attribution that leverages decoder attentions to directly cite supporting text spans or images.<n>We introduce two strategies for multimodal attribution: a raw image mode, which directly uses image patch attentions, and a caption-as-span mode, which substitutes images with generated captions to enable purely text-based alignment.
  arXiv  Detail & Related papers  (2026-01-23T02:01:43Z)
• Descriptive Image-Text Matching with Graded Contextual Similarity [41.048698396716354]
  We propose descriptive image-text matching, called DITM, to learn the graded contextual similarity between image and text.<n>We formulate the descriptiveness score of each sentence with cumulative term frequency-inverse document frequency (TF-IDF) to balance the pairwise similarity.<n>Our method leverages sentence descriptiveness to learn robust image-text matching in two key ways.
  arXiv  Detail & Related papers  (2025-05-15T06:21:00Z)
• Data-Efficient Generalization for Zero-shot Composed Image Retrieval [67.46975191141928]
  ZS-CIR aims to retrieve the target image based on a reference image and a text description without requiring in-distribution triplets for training.<n>One prevalent approach follows the vision-language pretraining paradigm that employs a mapping network to transfer the image embedding to a pseudo-word token in the text embedding space.<n>We propose a Data-efficient Generalization (DeG) framework, including two novel designs, namely, Textual Supplement (TS) module and Semantic-Set (S-Set)
  arXiv  Detail & Related papers  (2025-03-07T07:49:31Z)
• Language-Guided Visual Perception Disentanglement for Image Quality Assessment and Conditional Image Generation [48.642826318384294]
  Contrastive vision-language models, such as CLIP, have demonstrated excellent zero-shot capability across semantic recognition tasks.<n>This paper presents a new multimodal disentangled representation learning framework, which leverages disentangled text to guide image disentanglement.
  arXiv  Detail & Related papers  (2025-03-04T02:36:48Z)
• Evaluating Image Caption via Cycle-consistent Text-to-Image Generation [24.455344211552692]
  We propose CAMScore, a reference-free automatic evaluation metric for image captioning models.<n>To circumvent the aforementioned modality gap, CAMScore utilizes a text-to-image model to generate images from captions and subsequently evaluates these generated images against the original images.<n>Experiment results show that CAMScore achieves a superior correlation with human judgments compared to existing reference-based and reference-free metrics.
  arXiv  Detail & Related papers  (2025-01-07T06:35:34Z)
• A Novel Evaluation Framework for Image2Text Generation [15.10524860121122]
  We propose an evaluation framework rooted in a modern large language model (LLM) capable of image generation. A high similarity score suggests that the image captioning model has accurately generated textual descriptions. A low similarity score indicates discrepancies, revealing potential shortcomings in the model's performance.
  arXiv  Detail & Related papers  (2024-08-03T09:27:57Z)
• Noisy-Correspondence Learning for Text-to-Image Person Re-identification [50.07634676709067]
  We propose a novel Robust Dual Embedding method (RDE) to learn robust visual-semantic associations even with noisy correspondences. Our method achieves state-of-the-art results both with and without synthetic noisy correspondences on three datasets.
  arXiv  Detail & Related papers  (2023-08-19T05:34:13Z)
• Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training [33.78990448307792]
  Image-text retrieval is a central problem for understanding the semantic relationship between vision and language. Previous works either simply learn coarse-grained representations of the overall image and text, or elaborately establish the correspondence between image regions or pixels and text words. In this work, we address image-text retrieval from a novel perspective by combining coarse- and fine-grained representation learning into a unified framework.
  arXiv  Detail & Related papers  (2023-06-15T00:19:13Z)
• Collaborative Group: Composed Image Retrieval via Consensus Learning from Noisy Annotations [67.92679668612858]
  We propose the Consensus Network (Css-Net), inspired by the psychological concept that groups outperform individuals. Css-Net comprises two core components: (1) a consensus module with four diverse compositors, each generating distinct image-text embeddings; and (2) a Kullback-Leibler divergence loss that encourages learning of inter-compositor interactions. On benchmark datasets, particularly FashionIQ, Css-Net demonstrates marked improvements. Notably, it achieves significant recall gains, with a 2.77% increase in R@10 and 6.67% boost in R@50, underscoring its
  arXiv  Detail & Related papers  (2023-06-03T11:50:44Z)
• Transparent Human Evaluation for Image Captioning [70.03979566548823]
  We develop a rubric-based human evaluation protocol for image captioning models. We show that human-generated captions show substantially higher quality than machine-generated ones. We hope that this work will promote a more transparent evaluation protocol for image captioning.
  arXiv  Detail & Related papers  (2021-11-17T07:09:59Z)
• Contrastive Semantic Similarity Learning for Image Captioning Evaluation with Intrinsic Auto-encoder [52.42057181754076]
  Motivated by the auto-encoder mechanism and contrastive representation learning advances, we propose a learning-based metric for image captioning. We develop three progressive model structures to learn the sentence level representations. Experiment results show that our proposed method can align well with the scores generated from other contemporary metrics.
  arXiv  Detail & Related papers  (2021-06-29T12:27:05Z)
• Intrinsic Image Captioning Evaluation [53.51379676690971]
  We propose a learning based metrics for image captioning, which we call Intrinsic Image Captioning Evaluation(I2CE) Experiment results show that our proposed method can keep robust performance and give more flexible scores to candidate captions when encountered with semantic similar expression or less aligned semantics.
  arXiv  Detail & Related papers  (2020-12-14T08:36:05Z)
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching [69.34076386926984]
  Image-text matching plays a central role in bridging vision and language. Most existing approaches only rely on the image-text instance pair to learn their representations. We propose a Consensus-aware Visual-Semantic Embedding model to incorporate the consensus information.
  arXiv  Detail & Related papers  (2020-07-17T10:22:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.