VisKnow: Constructing Visual Knowledge Base for Object Understanding

• URL: http://arxiv.org/abs/2512.08221v1
• Date: Tue, 09 Dec 2025 04:00:25 GMT
• Title: VisKnow: Constructing Visual Knowledge Base for Object Understanding
• Authors: Ziwei Yao, Qiyang Wan, Ruiping Wang, Xilin Chen,
• Abstract summary: We propose the Visual Knowledge Base that structures multi-modal object knowledge as graphs, and present a construction framework named VisKnow.<n>As a specific case study, we construct AnimalKB, a structured animal knowledge base covering 406 animal categories.<n>A series of experiments showcase how AnimalKB enhances object-level visual tasks such as zero-shot recognition and fine-grained VQA.
• Score: 34.5763329787359
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Understanding objects is fundamental to computer vision. Beyond object recognition that provides only a category label as typical output, in-depth object understanding represents a comprehensive perception of an object category, involving its components, appearance characteristics, inter-category relationships, contextual background knowledge, etc. Developing such capability requires sufficient multi-modal data, including visual annotations such as parts, attributes, and co-occurrences for specific tasks, as well as textual knowledge to support high-level tasks like reasoning and question answering. However, these data are generally task-oriented and not systematically organized enough to achieve the expected understanding of object categories. In response, we propose the Visual Knowledge Base that structures multi-modal object knowledge as graphs, and present a construction framework named VisKnow that extracts multi-modal, object-level knowledge for object understanding. This framework integrates enriched aligned text and image-source knowledge with region annotations at both object and part levels through a combination of expert design and large-scale model application. As a specific case study, we construct AnimalKB, a structured animal knowledge base covering 406 animal categories, which contains 22K textual knowledge triplets extracted from encyclopedic documents, 420K images, and corresponding region annotations. A series of experiments showcase how AnimalKB enhances object-level visual tasks such as zero-shot recognition and fine-grained VQA, and serves as challenging benchmarks for knowledge graph completion and part segmentation. Our findings highlight the potential of automatically constructing visual knowledge bases to advance visual understanding and its practical applications. The project page is available at https://vipl-vsu.github.io/VisKnow.

Related papers

• Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learning [17.580250180523752]
  Open-domain visual entity recognition aims to identify and link entities depicted in images to a vast and evolving set of real-world concepts.<n>We propose a Knowledge-guided Contrastive Learning framework that combines both images and text descriptions into a shared semantic space.<n>Our experiments show that using visual, textual, and structured knowledge greatly improves accuracy, especially for rare and unseen entities.
  arXiv  Detail & Related papers  (2025-10-15T15:33:36Z)
• Augmented Commonsense Knowledge for Remote Object Grounding [67.30864498454805]
  We propose an augmented commonsense knowledge model (ACK) to leverage commonsense information as atemporal knowledge graph for improving agent navigation. ACK consists of knowledge graph-aware cross-modal and concept aggregation modules to enhance visual representation and visual-textual data alignment. We add a new pipeline for the commonsense-based decision-making process which leads to more accurate local action prediction.
  arXiv  Detail & Related papers  (2024-06-03T12:12:33Z)
• Object Attribute Matters in Visual Question Answering [15.705504296316576]
  We propose a novel VQA approach from the perspective of utilizing object attribute. The attribute fusion module constructs a multimodal graph neural network to fuse attributes and visual features through message passing. The better object-level visual-language alignment aids in understanding multimodal scenes, thereby improving the model's robustness.
  arXiv  Detail & Related papers  (2023-12-20T12:46:30Z)
• CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection [42.2847114428716]
  Task driven object detection aims to detect object instances suitable for affording a task in an image. Its challenge lies in object categories available for the task being too diverse to be limited to a closed set of object vocabulary for traditional object detection. We propose to explore fundamental affordances rather than object categories, i.e., common attributes that enable different objects to accomplish the same task.
  arXiv  Detail & Related papers  (2023-09-03T06:18:39Z)
• Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models [64.24227572048075]
  We propose a Knowledge-Aware Prompt Tuning (KAPT) framework for vision-language models. Our approach takes inspiration from human intelligence in which external knowledge is usually incorporated into recognizing novel categories of objects.
  arXiv  Detail & Related papers  (2023-08-22T04:24:45Z)
• Learning by Asking Questions for Knowledge-based Novel Object Recognition [64.55573343404572]
  In real-world object recognition, there are numerous object classes to be recognized. Conventional image recognition based on supervised learning can only recognize object classes that exist in the training data, and thus has limited applicability in the real world. Inspired by this, we study a framework for acquiring external knowledge through question generation that would help the model instantly recognize novel objects. Our pipeline consists of two components: the Object-based object recognition, and the Question Generator, which generates knowledge-aware questions to acquire novel knowledge.
  arXiv  Detail & Related papers  (2022-10-12T02:51:58Z)
• Contrastive Object Detection Using Knowledge Graph Embeddings [72.17159795485915]
  We compare the error statistics of the class embeddings learned from a one-hot approach with semantically structured embeddings from natural language processing or knowledge graphs. We propose a knowledge-embedded design for keypoint-based and transformer-based object detection architectures.
  arXiv  Detail & Related papers  (2021-12-21T17:10:21Z)
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge [59.87823082513752]
  This paper investigates the injection of knowledge from general-purpose knowledge bases (KBs) into vision-and-language transformers. We empirically study the relevance of various KBs to multiple tasks and benchmarks. The technique is model-agnostic and can expand the applicability of any vision-and-language transformer with minimal computational overhead.
  arXiv  Detail & Related papers  (2021-01-15T08:37:55Z)
• Look-into-Object: Self-supervised Structure Modeling for Object Recognition [71.68524003173219]
  We propose to "look into object" (explicitly yet intrinsically model the object structure) through incorporating self-supervisions. We show the recognition backbone can be substantially enhanced for more robust representation learning. Our approach achieves large performance gain on a number of benchmarks, including generic object recognition (ImageNet) and fine-grained object recognition tasks (CUB, Cars, Aircraft)
  arXiv  Detail & Related papers  (2020-03-31T12:22:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.