ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs

• URL: http://arxiv.org/abs/2609.00629v1
• Date: Tue, 01 Sep 2026 03:08:14 GMT
• Title: ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs
• Abstract summary: We propose a framework that exploits factual information from a knowledge graph via retrieval-augmented generation (RAG)<n>Specifically, our method alternates between answer generation and knowledge-graph retrieval, and controls the search using a correctness judgment.<n>We show experimentally that it improves the level of detail of artwork explanations and reduces the retrieval cost of external knowledge.
• Score: 51.31488026676001
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (LVLMs) achieve strong performance on image-grounded text generation and visual question answering. However, it remains difficult for them to comprehensively and accurately describe the factual relations among the entities and concepts associated with the objects depicted in an image. In this work, we propose a framework that efficiently exploits factual information from a knowledge graph via retrieval-augmented generation (RAG), with the goal of enabling LVLMs to generate detailed and accurate image explanations. Specifically, our method alternates between answer generation and knowledge-graph retrieval, and controls the search using a correctness judgment, thereby acquiring the necessary and sufficient factual information efficiently. We also construct a knowledge graph for the artwork domain (ExpArt-KG), in which the correspondence between images and entities is unambiguous. Applying the proposed method to this knowledge graph, we show experimentally that it improves the level of detail of artwork explanations and reduces the retrieval cost of external knowledge while maintaining generation quality comparable to that of iterating a fixed number of times.

Related papers

• CoVis: A Collaborative Framework for Fine-grained Graphic Visual Understanding [0.29127054707887967]
  CoVis is a collaborative framework for fine-grained visual understanding.<n>By designing and implementing a cascaded dual-layer segmentation network, it extracts as much knowledge as possible from an image.<n>It generates visual analytics for images, assisting observers in comprehending imagery from a more holistic perspective.
  arXiv  Detail & Related papers  (2024-11-27T21:38:04Z)
• Artwork Explanation in Large-scale Vision Language Models [32.645448509968226]
  Large-scale vision-language models (LVLMs) output text from images and instructions, demonstrating advanced capabilities in text generation and comprehension. We propose a new task: the artwork explanation generation task, along with its evaluation dataset and metric. It consists of two parts: generating explanations from both images and titles of artworks, and generating explanations using only images. Our findings indicate that LVLMs not only struggle with integrating language and visual information but also exhibit a more pronounced limitation in acquiring knowledge from images alone.
  arXiv  Detail & Related papers  (2024-02-29T19:01:03Z)
• KGrEaT: A Framework to Evaluate Knowledge Graphs via Downstream Tasks [1.8722948221596285]
  KGrEaT is a framework to estimate the quality of knowledge graphs via actual downstream tasks like classification, clustering, or recommendation. The framework takes a knowledge graph as input, automatically maps it to the datasets to be evaluated on, and computes performance metrics for the defined tasks.
  arXiv  Detail & Related papers  (2023-08-21T07:43:10Z)
• Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality [50.48859793121308]
  Contrastively trained vision-language models have achieved remarkable progress in vision and language representation learning. Recent research has highlighted severe limitations in their ability to perform compositional reasoning over objects, attributes, and relations.
  arXiv  Detail & Related papers  (2023-05-23T08:28:38Z)
• SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning [61.57887011165744]
  multimodal Transformers have made great progress in the task of Visual Commonsense Reasoning. We propose a Scene Graph Enhanced Image-Text Learning framework to incorporate visual scene graphs in commonsense reasoning.
  arXiv  Detail & Related papers  (2021-12-16T03:16:30Z)
• Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph [96.95815946327079]
  It is difficult to learn the association between named entities and visual cues due to the long-tail distribution of named entities. We propose a novel approach that constructs a multi-modal knowledge graph to associate the visual objects with named entities.
  arXiv  Detail & Related papers  (2021-07-26T05:50:41Z)
• Learning semantic Image attributes using Image recognition and knowledge graph embeddings [0.3222802562733786]
  We propose a shared learning approach to learn semantic attributes of images by combining a knowledge graph embedding model with the recognized attributes of images. The proposed approach is a step towards bridging the gap between frameworks which learn from large amounts of data and frameworks which use a limited set of predicates to infer new knowledge.
  arXiv  Detail & Related papers  (2020-09-12T15:18:48Z)
• ENT-DESC: Entity Description Generation by Exploring Knowledge Graph [53.03778194567752]
  In practice, the input knowledge could be more than enough, since the output description may only cover the most significant knowledge. We introduce a large-scale and challenging dataset to facilitate the study of such a practical scenario in KG-to-text. We propose a multi-graph structure that is able to represent the original graph information more comprehensively.
  arXiv  Detail & Related papers  (2020-04-30T14:16:19Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.