PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning

• URL: http://arxiv.org/abs/2303.08389v1
• Date: Wed, 15 Mar 2023 06:37:26 GMT
• Title: PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning
• Abstract summary: Perturbation Robust Multi-Lingual CLIPScore(PR-MCS) is a novel reference-free image captioning metric applicable to multiple languages. PR-MCS significantly outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five languages.
• Score: 30.150484615377497
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vulnerability to lexical perturbation is a critical weakness of automatic evaluation metrics for image captioning. This paper proposes Perturbation Robust Multi-Lingual CLIPScore(PR-MCS), which exhibits robustness to such perturbations, as a novel reference-free image captioning metric applicable to multiple languages. To achieve perturbation robustness, we fine-tune the text encoder of CLIP with our language-agnostic method to distinguish the perturbed text from the original text. To verify the robustness of PR-MCS, we introduce a new fine-grained evaluation dataset consisting of detailed captions, critical objects, and the relationships between the objects for 3, 000 images in five languages. In our experiments, PR-MCS significantly outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five languages, proving that PR-MCS is highly robust to lexical perturbations.

Related papers

• Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models [7.833222732846266]
  Optical character recognition (OCR) and multilingual text understanding remain major failure modes of multimodal large language models (MLLMs)<n>We present an OCR-aware multilingual multimodal training framework that combines large-scale synthetic OCR-to-translation data generation, OCR-aware supervised fine-tuning, and structured visual chain-of-thought.<n>Using a LLaMA-based multimodal architecture, the proposed framework substantially improves OCR completeness, multilingual translation accuracy, and robustness under degraded visual conditions.
  arXiv  Detail & Related papers  (2026-05-13T14:16:39Z)
• ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs [84.09282931360089]
  ICBench is a large-scale image captioning benchmark that covers 12 content categories and consists of both short and long captions on 2K images.<n>We conduct extensive human subjective studies to obtain mean opinion scores (MOSs) across fine-grained evaluation dimensions.<n>We propose an automated evaluation metric, textbfITIScore, based on an image-to-text-to-image framework, which measures caption quality through reconstruction consistency.
  arXiv  Detail & Related papers  (2026-04-04T15:33:07Z)
• Evaluating Extremely Low-Resource Machine Translation: A Comparative Study of ChrF++ and BLEU Metrics [69.2321983942375]
  This work presents a comparative analysis of BLEU, an n-gram-based metric, and ChrF++, a character-based metric, for MT evaluation in ELRL settings.<n>We examine how each metric responds to translation artifacts, including hallucinations, repetition, source-text copying, and diacritic (textitmatra) variations across three ELRLs: Magahi, Bhojpuri, and Chhattisgarhi.<n>While recent work often relies solely on ChrF++, our findings show that BLEU, despite its lower absolute scores, provides complementary lexical-precision insights that improve interpretability.
  arXiv  Detail & Related papers  (2026-02-19T14:56:42Z)
• Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Dataset [41.02122939888977]
  This study proposes a cross-native evaluation of Transformer-based vision and language models for Brazilian Portuguese IC.<n>We use a version of Flickr30K comprised of captions manually created by native Brazilian Portuguese speakers and compare it to a version with captions automatically translated from English to Portuguese.<n>Our findings show that Swin-DistilBERTimbau consistently outperforms other models, demonstrating strong generalization across datasets.
  arXiv  Detail & Related papers  (2026-01-30T23:12:03Z)
• On the Brittleness of CLIP Text Encoders [3.482579981426874]
  We present a systematic analysis of the effect of multiple classes of non-semantic query perturbations in a multimedia information retrieval scenario.<n>We find that syntactic and semantic perturbations drive the largest instabilities, while brittleness is concentrated in trivial surface edits such as punctuation and case.
  arXiv  Detail & Related papers  (2025-11-06T10:33:55Z)
• Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach [51.95266411355865]
  Autoregressive language models are vulnerable to orthographic attacks.<n>This vulnerability stems from the out-of-vocabulary issue inherent in subword tokenizers and their embeddings.<n>We propose a pixel-based generative language model that replaces the text-based embeddings with pixel-based representations by rendering words as individual images.
  arXiv  Detail & Related papers  (2025-08-28T20:48:38Z)
• Ultrasound Report Generation with Multimodal Large Language Models for Standardized Texts [15.349894506969074]
  We propose a unified framework for multi-organ and multilingual US report generation.<n>Method achieves consistent and clinically accurate text generation across organ sites and languages.
  arXiv  Detail & Related papers  (2025-05-13T08:27:01Z)
• CoLLM: A Large Language Model for Composed Image Retrieval [76.29725148964368]
  Composed Image Retrieval (CIR) is a complex task that aims to retrieve images based on a multimodal query. We present CoLLM, a one-stop framework that generates triplets on-the-fly from image-caption pairs. We leverage Large Language Models (LLMs) to generate joint embeddings of reference images and modification texts.
  arXiv  Detail & Related papers  (2025-03-25T17:59:50Z)
• PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training [56.172959986096316]
  This paper aims to address the challenge of hallucinations in Multimodal Large Language Models (MLLMs) HalFscore is a novel metric built upon the language graph and is designed to evaluate both the accuracy and completeness of dense captions at a granular level. PerturboLLaVA significantly improves the fidelity of generated captions, outperforming existing approaches in handling multimodal hallucinations.
  arXiv  Detail & Related papers  (2025-03-09T07:07:03Z)
• Evaluating Robustness of LLMs in Question Answering on Multilingual Noisy OCR Data [17.20084584886653]
  We conduct a comprehensive analysis of how OCR-induced noise affects the performance of Multilingual QA Systems.<n>To support this analysis, we introduce a multilingual QA dataset MultiOCR-QA, comprising 50K question-answer pairs across three languages.<n>The dataset is curated from OCR-ed historical documents, which include different levels and types of OCR noise.<n>We then evaluate how different state-of-the-art Large Language models (LLMs) perform under different error conditions, focusing on three major OCR error types.
  arXiv  Detail & Related papers  (2025-02-24T02:16:37Z)
• Deciphering the Underserved: Benchmarking LLM OCR for Low-Resource Scripts [0.0]
  This study investigates the potential of Large Language Models (LLMs), particularly GPT-4o, for Optical Character Recognition (OCR) in low-resource scripts such as Urdu, Albanian, and Tajik. Using a meticulously curated dataset of 2,520 images incorporating controlled variations in text length, font size, background color, and blur, the research simulates diverse real-world challenges.
  arXiv  Detail & Related papers  (2024-12-20T18:05:22Z)
• Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage [50.84150600032693]
  Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations. We propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions. Our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V.
  arXiv  Detail & Related papers  (2024-12-20T01:37:22Z)
• BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues [47.213906345208315]
  We propose BRIDGE, a new learnable and reference-free image captioning metric. Our proposal achieves state-of-the-art results compared to existing reference-free evaluation scores.
  arXiv  Detail & Related papers  (2024-07-29T18:00:17Z)
• Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences [80.54979242912944]
  This paper introduces Mementos, a new benchmark designed to assess MLLMs' sequential image reasoning abilities. We find that MLLMs struggle to accurately describe dynamic information about given image sequences, often leading to hallucinations/misrepresentations of objects.
  arXiv  Detail & Related papers  (2024-01-19T07:10:13Z)
• LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models [45.672539931681065]
  We propose a multi-level interaction paradigm for training lightweight CLIP models. An auxiliary fusion module injecting unmasked image embedding into masked text embedding is proposed.
  arXiv  Detail & Related papers  (2023-12-01T15:54:55Z)
• MLLMs-Augmented Visual-Language Representation Learning [70.5293060238008]
  We demonstrate that Multi-modal Large Language Models (MLLMs) can enhance visual-language representation learning. Our approach is simple, utilizing MLLMs to extend multiple diverse captions for each image. We propose "text shearing" to maintain the quality and availability of extended captions.
  arXiv  Detail & Related papers  (2023-11-30T18:05:52Z)
• Speculative Contrastive Decoding [55.378200871224074]
  Large language models(LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias. Inspired by speculative decoding and contrastive decoding, we introduce Speculative Contrastive Decoding(SCD), a straightforward yet powerful decoding approach.
  arXiv  Detail & Related papers  (2023-11-15T14:15:30Z)
• CLAIR: Evaluating Image Captions with Large Language Models [69.46906537973518]
  We propose CLAIR, a novel method to evaluate machine-generated image captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Clair provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score.
  arXiv  Detail & Related papers  (2023-10-19T17:59:01Z)
• Sieve: Multimodal Dataset Pruning Using Image Captioning Models [11.362835828985494]
  Vision-Language Models (VLMs) are pretrained on large, diverse, and noisy web-crawled datasets. We argue that this approach suffers from multiple limitations including false positives and negatives due to CLIP's pretraining on noisy labels. We propose a pruning signal, Sieve, that employs synthetic captions generated by image-captioning models pretrained on small, diverse, and well-aligned image-text pairs.
  arXiv  Detail & Related papers  (2023-10-03T14:53:53Z)
• An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics [6.137805050050414]
  We evaluate CLIPScore, UMIC, and PAC-S in scenarios that require distinguishing between two captions with high lexical overlap but very different meanings. Our findings reveal that despite their high correlation with human judgments, CLIPScore, UMIC, and PAC-S struggle to identify fine-grained errors.
  arXiv  Detail & Related papers  (2023-05-24T10:36:12Z)
• Transparent Human Evaluation for Image Captioning [70.03979566548823]
  We develop a rubric-based human evaluation protocol for image captioning models. We show that human-generated captions show substantially higher quality than machine-generated ones. We hope that this work will promote a more transparent evaluation protocol for image captioning.
  arXiv  Detail & Related papers  (2021-11-17T07:09:59Z)
• Intrinsic Image Captioning Evaluation [53.51379676690971]
  We propose a learning based metrics for image captioning, which we call Intrinsic Image Captioning Evaluation(I2CE) Experiment results show that our proposed method can keep robust performance and give more flexible scores to candidate captions when encountered with semantic similar expression or less aligned semantics.
  arXiv  Detail & Related papers  (2020-12-14T08:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.