M^2ConceptBase: A Fine-Grained Aligned Concept-Centric Multimodal Knowledge Base

• URL: http://arxiv.org/abs/2312.10417v2
• Date: Thu, 1 Aug 2024 08:03:48 GMT
• Title: M^2ConceptBase: A Fine-Grained Aligned Concept-Centric Multimodal Knowledge Base
• Authors: Zhiwei Zha, Jiaan Wang, Zhixu Li, Xiangru Zhu, Wei Song, Yanghua Xiao,
• Abstract summary: We introduce M2ConceptBase, the first concept-centric multimodal knowledge base (MMKB) We propose a context-aware approach to align concept-image and concept-description pairs using context information from image-text datasets. Human studies confirm more than 95% alignment accuracy, underscoring its quality.
• Score: 61.53959791360333
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal knowledge bases (MMKBs) provide cross-modal aligned knowledge crucial for multimodal tasks. However, the images in existing MMKBs are generally collected for entities in encyclopedia knowledge graphs. Therefore, detailed groundings of visual semantics with linguistic concepts are lacking, which are essential for the visual concept cognition ability of multimodal models. Addressing this gap, we introduce M^2ConceptBase, the first concept-centric MMKB. M^2ConceptBase models concepts as nodes with associated images and detailed textual descriptions. We propose a context-aware multimodal symbol grounding approach to align concept-image and concept-description pairs using context information from image-text datasets. Comprising 951K images and 152K concepts, M^2ConceptBase links each concept to an average of 6.27 images and a single description, ensuring comprehensive visual and textual semantics. Human studies confirm more than 95% alignment accuracy, underscoring its quality. Additionally, our experiments demonstrate that M^2ConceptBase significantly enhances VQA model performance on the OK-VQA task. M^2ConceptBase also substantially improves the fine-grained concept understanding capabilities of multimodal large language models through retrieval augmentation in two concept-related tasks, highlighting its value.

Related papers

• CusConcept: Customized Visual Concept Decomposition with Diffusion Models [13.95568624067449]
  We propose a two-stage framework, CusConcept, to extract customized visual concept embedding vectors. In the first stage, CusConcept employs a vocabularies-guided concept decomposition mechanism. In the second stage, joint concept refinement is performed to enhance the fidelity and quality of generated images.
  arXiv  Detail & Related papers  (2024-10-01T04:41:44Z)
• ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty [52.15933752463479]
  ConceptMix is a scalable, controllable, and customizable benchmark. It automatically evaluates compositional generation ability of Text-to-Image (T2I) models. It reveals that the performance of several models, especially open models, drops dramatically with increased k.
  arXiv  Detail & Related papers  (2024-08-26T15:08:12Z)
• A Concept-Based Explainability Framework for Large Multimodal Models [52.37626977572413]
  We propose a dictionary learning based approach, applied to the representation of tokens. We show that these concepts are well semantically grounded in both vision and text. We show that the extracted multimodal concepts are useful to interpret representations of test samples.
  arXiv  Detail & Related papers  (2024-06-12T10:48:53Z)
• Visual Concept-driven Image Generation with Text-to-Image Diffusion Model [65.96212844602866]
  Text-to-image (TTI) models have demonstrated impressive results in generating high-resolution images of complex scenes. Recent approaches have extended these methods with personalization techniques that allow them to integrate user-illustrated concepts. However, the ability to generate images with multiple interacting concepts, such as human subjects, as well as concepts that may be entangled in one, or across multiple, image illustrations remains illusive. We propose a concept-driven TTI personalization framework that addresses these core challenges.
  arXiv  Detail & Related papers  (2024-02-18T07:28:37Z)
• NEUCORE: Neural Concept Reasoning for Composed Image Retrieval [16.08214739525615]
  We propose a NEUral COncept REasoning model which incorporates multi-modal concept alignment and progressive multimodal fusion over aligned concepts. Our proposed approach is evaluated on three datasets and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2023-10-02T17:21:25Z)
• Create Your World: Lifelong Text-to-Image Diffusion [75.14353789007902]
  We propose Lifelong text-to-image Diffusion Model (L2DM) to overcome knowledge "catastrophic forgetting" for the past encountered concepts. In respect of knowledge "catastrophic forgetting", our L2DM framework devises a task-aware memory enhancement module and a elastic-concept distillation module. Our model can generate more faithful image across a range of continual text prompts in terms of both qualitative and quantitative metrics.
  arXiv  Detail & Related papers  (2023-09-08T16:45:56Z)
• ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models [79.10890337599166]
  We introduce ConceptBed, a large-scale dataset that consists of 284 unique visual concepts and 33K composite text prompts. We evaluate visual concepts that are either objects, attributes, or styles, and also evaluate four dimensions of compositionality: counting, attributes, relations, and actions. Our results point to a trade-off between learning the concepts and preserving the compositionality which existing approaches struggle to overcome.
  arXiv  Detail & Related papers  (2023-06-07T18:00:38Z)
• Automatic Modeling of Social Concepts Evoked by Art Images as Multimodal Frames [1.4502611532302037]
  Social concepts referring to non-physical objects are powerful tools to describe, index, and query the content of visual data. We propose a software approach to represent social concepts as multimodal frames, by integrating multisensory data. Our method focuses on the extraction, analysis, and integration of multimodal features from visual art material tagged with the concepts of interest.
  arXiv  Detail & Related papers  (2021-10-14T14:50:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.