Language Model Analysis for Ontology Subsumption Inference

• URL: http://arxiv.org/abs/2302.06761v3
• Date: Mon, 8 May 2023 17:21:40 GMT
• Title: Language Model Analysis for Ontology Subsumption Inference
• Authors: Yuan He, Jiaoyan Chen, Ernesto Jim\'enez-Ruiz, Hang Dong, Ian Horrocks
• Abstract summary: We investigate whether pre-trained language models (LMs) can function as knowledge bases (KBs) We propose OntoLAMA, a set of inference-based probing tasks and datasets from subsumption axioms involving both atomic and complex concepts. We conduct extensive experiments on different domains and scales, and our results demonstrate that LMs encode relatively less background knowledge of Subsumption Inference (SI) than traditional Natural Language Inference (NLI) We will open-source our code and datasets.
• Score: 37.00562636991463
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Investigating whether pre-trained language models (LMs) can function as knowledge bases (KBs) has raised wide research interests recently. However, existing works focus on simple, triple-based, relational KBs, but omit more sophisticated, logic-based, conceptualised KBs such as OWL ontologies. To investigate an LM's knowledge of ontologies, we propose OntoLAMA, a set of inference-based probing tasks and datasets from ontology subsumption axioms involving both atomic and complex concepts. We conduct extensive experiments on ontologies of different domains and scales, and our results demonstrate that LMs encode relatively less background knowledge of Subsumption Inference (SI) than traditional Natural Language Inference (NLI) but can improve on SI significantly when a small number of samples are given. We will open-source our code and datasets.

Related papers

• Knowledge Graph Structure as Prompt: Improving Small Language Models Capabilities for Knowledge-based Causal Discovery [10.573861741540853]
  KG Structure as Prompt is a novel approach for integrating structural information from a knowledge graph, such as common neighbor nodes and metapaths, into prompt-based learning. Experimental results on three types of biomedical and open-domain datasets under few-shot settings demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-07-26T14:07:00Z)
• Large Language Models as Reliable Knowledge Bases? [60.25969380388974]
  Large Language Models (LLMs) can be viewed as potential knowledge bases (KBs) This study defines criteria that a reliable LLM-as-KB should meet, focusing on factuality and consistency. strategies like ICL and fine-tuning are unsuccessful at making LLMs better KBs.
  arXiv  Detail & Related papers  (2024-07-18T15:20:18Z)
• Chain-of-Knowledge: Integrating Knowledge Reasoning into Large Language Models by Learning from Knowledge Graphs [55.317267269115845]
  Chain-of-Knowledge (CoK) is a comprehensive framework for knowledge reasoning. CoK includes methodologies for both dataset construction and model learning. We conduct extensive experiments with KnowReason.
  arXiv  Detail & Related papers  (2024-06-30T10:49:32Z)
• SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models [70.01883340129204]
  spatial reasoning is a crucial component of both biological and artificial intelligence. We present a comprehensive study of the capability of current state-of-the-art large language models (LLMs) on spatial reasoning.
  arXiv  Detail & Related papers  (2024-06-07T01:06:34Z)
• Quantitative knowledge retrieval from large language models [4.155711233354597]
  Large language models (LLMs) have been extensively studied for their abilities to generate convincing natural language sequences. This paper explores the feasibility of LLMs as a mechanism for quantitative knowledge retrieval to aid data analysis tasks.
  arXiv  Detail & Related papers  (2024-02-12T16:32:37Z)
• Evaluating Neural Language Models as Cognitive Models of Language Acquisition [4.779196219827507]
  We argue that some of the most prominent benchmarks for evaluating the syntactic capacities of neural language models may not be sufficiently rigorous. When trained on small-scale data modeling child language acquisition, the LMs can be readily matched by simple baseline models. We conclude with suggestions for better connecting LMs with the empirical study of child language acquisition.
  arXiv  Detail & Related papers  (2023-10-31T00:16:17Z)
• Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach [50.125704610228254]
  Large Language Models (LLMs) have not only exhibited exceptional performance across various tasks, but also demonstrated sparks of intelligence. Recent studies have focused on assessing their capabilities on human exams and revealed their impressive competence in different domains. We conduct an evaluation using MoocRadar, a meticulously annotated human test dataset based on Bloom taxonomy.
  arXiv  Detail & Related papers  (2023-10-12T09:55:45Z)
• Data Distribution Bottlenecks in Grounding Language Models to Knowledge Bases [9.610231090476857]
  Language models (LMs) have already demonstrated remarkable abilities in understanding and generating both natural and formal language. This paper is an experimental investigation aimed at uncovering the challenges that LMs encounter when tasked with knowledge base question answering (KBQA) Our experiments reveal that even when employed with our proposed data augmentation techniques, advanced small and large language models exhibit poor performance in various dimensions.
  arXiv  Detail & Related papers  (2023-09-15T12:06:45Z)
• Information Extraction in Low-Resource Scenarios: Survey and Perspective [56.5556523013924]
  Information Extraction seeks to derive structured information from unstructured texts. This paper presents a review of neural approaches to low-resource IE from emphtraditional and emphLLM-based perspectives.
  arXiv  Detail & Related papers  (2022-02-16T13:44:00Z)
• Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries [35.57443199012129]
  Pretrained language models have been suggested as a possible alternative or complement to structured knowledge bases. Here, we formulate two basic requirements for treating LMs as KBs. We explore three entity representations that allow LMs to represent millions of entities and present a detailed case study on paraphrased querying of world knowledge in LMs.
  arXiv  Detail & Related papers  (2020-08-20T15:39:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.