Evaluating Image Review Ability of Vision Language Models

• URL: http://arxiv.org/abs/2402.12121v1
• Date: Mon, 19 Feb 2024 13:16:10 GMT
• Title: Evaluating Image Review Ability of Vision Language Models
• Authors: Shigeki Saito, Kazuki Hayashi, Yusuke Ide, Yusuke Sakai, Kazuma Onishi, Toma Suzuki, Seiji Gobara, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe
• Abstract summary: This paper explores the use of large-scale vision language models (LVLMs) to generate review texts for images. The ability of LVLMs to review images is not fully understood, highlighting the need for a methodical evaluation of their review abilities.
• Score: 25.846728716526766
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large-scale vision language models (LVLMs) are language models that are capable of processing images and text inputs by a single model. This paper explores the use of LVLMs to generate review texts for images. The ability of LVLMs to review images is not fully understood, highlighting the need for a methodical evaluation of their review abilities. Unlike image captions, review texts can be written from various perspectives such as image composition and exposure. This diversity of review perspectives makes it difficult to uniquely determine a single correct review for an image. To address this challenge, we introduce an evaluation method based on rank correlation analysis, in which review texts are ranked by humans and LVLMs, then, measures the correlation between these rankings. We further validate this approach by creating a benchmark dataset aimed at assessing the image review ability of recent LVLMs. Our experiments with the dataset reveal that LVLMs, particularly those with proven superiority in other evaluative contexts, excel at distinguishing between high-quality and substandard image reviews.

Related papers

• TypeScore: A Text Fidelity Metric for Text-to-Image Generative Models [39.06617653124486]
  We introduce a new evaluation framework called TypeScore to assess a model's ability to generate images with high-fidelity embedded text. Our proposed metric demonstrates greater resolution than CLIPScore to differentiate popular image generation models.
  arXiv  Detail & Related papers  (2024-11-02T07:56:54Z)
• Beyond Coarse-Grained Matching in Video-Text Retrieval [50.799697216533914]
  We introduce a new approach for fine-grained evaluation. Our approach can be applied to existing datasets by automatically generating hard negative test captions. Experiments on our fine-grained evaluations demonstrate that this approach enhances a model's ability to understand fine-grained differences.
  arXiv  Detail & Related papers  (2024-10-16T09:42:29Z)
• A Novel Evaluation Framework for Image2Text Generation [15.10524860121122]
  We propose an evaluation framework rooted in a modern large language model (LLM) capable of image generation. A high similarity score suggests that the image captioning model has accurately generated textual descriptions. A low similarity score indicates discrepancies, revealing potential shortcomings in the model's performance.
  arXiv  Detail & Related papers  (2024-08-03T09:27:57Z)
• FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction [66.98008357232428]
  We propose FineMatch, a new aspect-based fine-grained text and image matching benchmark. FineMatch focuses on text and image mismatch detection and correction. We show that models trained on FineMatch demonstrate enhanced proficiency in detecting fine-grained text and image mismatches.
  arXiv  Detail & Related papers  (2024-04-23T03:42:14Z)
• Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction [27.00018283430169]
  This paper presents VisCE$2$, a vision language model-based caption evaluation method. Our method focuses on visual context, which refers to the detailed content of images, including objects, attributes, and relationships.
  arXiv  Detail & Related papers  (2024-02-28T01:29:36Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• Q-Bench+: A Benchmark for Multi-modal Foundation Models on Low-level Vision from Single Images to Pairs [71.07108539262721]
  We design benchmark settings to emulate human language responses related to low-level vision. We extend the low-level perception-related question-answering and description evaluations of MLLMs from single images to image pairs. We demonstrate that several MLLMs have decent low-level visual competencies on single images, but only GPT-4V exhibits higher accuracy on pairwise comparisons than humans.
  arXiv  Detail & Related papers  (2024-02-11T06:44:11Z)
• Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels [95.44077384918725]
  We propose to teach large multi-modality models (LMMs) with text-defined rating levels instead of scores. The proposed Q-Align achieves state-of-the-art performance on image quality assessment (IQA), image aesthetic assessment (IAA) and video quality assessment (VQA) tasks.
  arXiv  Detail & Related papers  (2023-12-28T16:10:25Z)
• Prompt-based Learning for Unpaired Image Captioning [86.44188293709307]
  Unpaired Image Captioning (UIC) has been developed to learn image descriptions from unaligned vision-language sample pairs. Recent successes of Vision-Language Pre-Trained Models (VL-PTMs) have triggered the development of prompt-based learning. We present in this paper a novel scheme based on prompt to train the UIC model, making best use of the powerful generalization ability.
  arXiv  Detail & Related papers  (2022-05-26T03:13:43Z)
• Deep Learning Approaches on Image Captioning: A Review [0.5852077003870417]
  Image captioning aims to generate natural language descriptions for visual content in the form of still images. Deep learning and vision-language pre-training techniques have revolutionized the field, leading to more sophisticated methods and improved performance. We address the challenges faced in this field by emphasizing issues such as object hallucination, missing context, illumination conditions, contextual understanding, and referring expressions. We identify several potential future directions for research in this area, which include tackling the information misalignment problem between image and text modalities, mitigating dataset bias, incorporating vision-language pre-training methods to enhance caption generation, and developing improved evaluation tools to accurately
  arXiv  Detail & Related papers  (2022-01-31T00:39:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.