Towards Ontology-Enhanced Representation Learning for Large Language Models

• URL: http://arxiv.org/abs/2405.20527v1
• Date: Thu, 30 May 2024 23:01:10 GMT
• Title: Towards Ontology-Enhanced Representation Learning for Large Language Models
• Authors: Francesco Ronzano, Jay Nanavati,
• Abstract summary: We propose a novel approach to improve an embedding-Large Language Model (embedding-LLM) of interest by infusing knowledge by a reference ontology. The linguistic information (i.e. concept synonyms and descriptions) and structural information (i.e. is-a relations) are utilized to compile a comprehensive set of concept definitions. These concept definitions are then employed to fine-tune the target embedding-LLM using a contrastive learning framework.
• Score: 0.18416014644193066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Taking advantage of the widespread use of ontologies to organise and harmonize knowledge across several distinct domains, this paper proposes a novel approach to improve an embedding-Large Language Model (embedding-LLM) of interest by infusing the knowledge formalized by a reference ontology: ontological knowledge infusion aims at boosting the ability of the considered LLM to effectively model the knowledge domain described by the infused ontology. The linguistic information (i.e. concept synonyms and descriptions) and structural information (i.e. is-a relations) formalized by the ontology are utilized to compile a comprehensive set of concept definitions, with the assistance of a powerful generative LLM (i.e. GPT-3.5-turbo). These concept definitions are then employed to fine-tune the target embedding-LLM using a contrastive learning framework. To demonstrate and evaluate the proposed approach, we utilize the biomedical disease ontology MONDO. The results show that embedding-LLMs enhanced by ontological disease knowledge exhibit an improved capability to effectively evaluate the similarity of in-domain sentences from biomedical documents mentioning diseases, without compromising their out-of-domain performance.

Related papers

• Enhancing Explainability in Multimodal Large Language Models Using Ontological Context [0.0]
  We propose a new framework that combines ontology with MLLMs to classify images of plant diseases. Ensuring that the model accurately uses the concepts describing the disease is crucial in domain-specific applications. Our framework offers a new direction for synergizing and MLLMs, supported by an empirical study using different well-known MLLMs.
  arXiv  Detail & Related papers  (2024-09-27T13:44:55Z)
• Document-level Clinical Entity and Relation Extraction via Knowledge Base-Guided Generation [0.869967783513041]
  We leverage the Unified Medical Language System (UMLS) knowledge base to accurately identify medical concepts. Our framework selects UMLS concepts relevant to the text and combines them with prompts to guide language models in extracting entities.
  arXiv  Detail & Related papers  (2024-07-13T22:45:46Z)
• Knowledge-enhanced Visual-Language Pretraining for Computational Pathology [68.6831438330526]
  We consider the problem of visual representation learning for computational pathology, by exploiting large-scale image-text pairs gathered from public resources. We curate a pathology knowledge tree that consists of 50,470 informative attributes for 4,718 diseases requiring pathology diagnosis from 32 human tissues.
  arXiv  Detail & Related papers  (2024-04-15T17:11:25Z)
• MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning [48.97640824497327]
  We propose a novel framework leveraging domain-specific medical knowledge as guiding signals to integrate language information into the visual domain through image-text contrastive learning. Our model includes global contrastive learning with our designed divergence encoder, local token-knowledge-patch alignment contrastive learning, and knowledge-guided category-level contrastive learning with expert knowledge. Notably, MLIP surpasses state-of-the-art methods even with limited annotated data, highlighting the potential of multimodal pre-training in advancing medical representation learning.
  arXiv  Detail & Related papers  (2024-02-03T05:48:50Z)
• Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs [54.223394825528665]
  We develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models. We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low.
  arXiv  Detail & Related papers  (2023-12-21T14:26:57Z)
• Can Large Language Models Augment a Biomedical Ontology with missing Concepts and Relations? [1.1060425537315088]
  We propose a method that uses semantic interactions with an LLM to analyze clinical practice guidelines. Our initial experimentation with the prompts yielded promising results given a manually generated gold standard.
  arXiv  Detail & Related papers  (2023-11-12T14:20:55Z)
• Exploring the In-context Learning Ability of Large Language Model for Biomedical Concept Linking [4.8882241537236455]
  This research investigates a method that exploits the in-context learning capabilities of large models for biomedical concept linking. The proposed approach adopts a two-stage retrieve-and-rank framework. It achieved an accuracy of 90.% in BC5CDR disease entity normalization and 94.7% in chemical entity normalization.
  arXiv  Detail & Related papers  (2023-07-03T16:19:50Z)
• EBOCA: Evidences for BiOmedical Concepts Association Ontology [55.41644538483948]
  This paper proposes EBOCA, an ontology that describes (i) biomedical domain concepts and associations between them, and (ii) evidences supporting these associations. Test data coming from a subset of DISNET and automatic association extractions from texts has been transformed to create a Knowledge Graph that can be used in real scenarios.
  arXiv  Detail & Related papers  (2022-08-01T18:47:03Z)
• A Meta-embedding-based Ensemble Approach for ICD Coding Prediction [64.42386426730695]
  International Classification of Diseases (ICD) are the de facto codes used globally for clinical coding. These codes enable healthcare providers to claim reimbursement and facilitate efficient storage and retrieval of diagnostic information. Our proposed approach enhances the performance of neural models by effectively training word vectors using routine medical data as well as external knowledge from scientific articles.
  arXiv  Detail & Related papers  (2021-02-26T17:49:58Z)
• UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus [73.86656026386038]
  We introduce UmlsBERT, a contextual embedding model that integrates domain knowledge during the pre-training process. By applying these two strategies, UmlsBERT can encode clinical domain knowledge into word embeddings and outperform existing domain-specific models.
  arXiv  Detail & Related papers  (2020-10-20T15:56:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.