Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge
of Pre-trained Language Models
- URL: http://arxiv.org/abs/2110.08173v1
- Date: Fri, 15 Oct 2021 16:00:11 GMT
- Title: Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge
of Pre-trained Language Models
- Authors: Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su, Charlotte
Collins, Nigel Collier
- Abstract summary: We release a well-curated biomedical knowledge probing benchmark, MedLAMA, based on the Unified Medical Language System (UMLS) Metathesaurus.
We test a wide spectrum of state-of-the-art PLMs and probing approaches on our benchmark, reaching at most 3% of acc@10.
We propose Contrastive-Probe, a novel self-supervised contrastive probing approach, that adjusts the underlying PLMs without using any probing data.
- Score: 16.535312449449165
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Knowledge probing is crucial for understanding the knowledge transfer
mechanism behind the pre-trained language models (PLMs). Despite the growing
progress of probing knowledge for PLMs in the general domain, specialised areas
such as biomedical domain are vastly under-explored. To catalyse the research
in this direction, we release a well-curated biomedical knowledge probing
benchmark, MedLAMA, which is constructed based on the Unified Medical Language
System (UMLS) Metathesaurus. We test a wide spectrum of state-of-the-art PLMs
and probing approaches on our benchmark, reaching at most 3% of acc@10. While
highlighting various sources of domain-specific challenges that amount to this
underwhelming performance, we illustrate that the underlying PLMs have a higher
potential for probing tasks. To achieve this, we propose Contrastive-Probe, a
novel self-supervised contrastive probing approach, that adjusts the underlying
PLMs without using any probing data. While Contrastive-Probe pushes the acc@10
to 28%, the performance gap still remains notable. Our human expert evaluation
suggests that the probing performance of our Contrastive-Probe is still
under-estimated as UMLS still does not include the full spectrum of factual
knowledge. We hope MedLAMA and Contrastive-Probe facilitate further
developments of more suited probing techniques for this domain.
Related papers
- On-the-fly Definition Augmentation of LLMs for Biomedical NER [28.02028191114401]
LLMs struggle on biomedical NER tasks due to specialized terminology and lack of training data.
We develop a new knowledge augmentation approach which incorporates definitions of relevant concepts on-the-fly.
We find that careful prompting strategies also improve LLM performance, allowing them to outperform fine-tuned language models in few-shot settings.
arXiv Detail & Related papers (2024-03-29T20:59:27Z) - Diversifying Knowledge Enhancement of Biomedical Language Models using
Adapter Modules and Knowledge Graphs [54.223394825528665]
We develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models.
We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT.
We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low.
arXiv Detail & Related papers (2023-12-21T14:26:57Z) - Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case
Study in Medicine [89.46836590149883]
We build on a prior study of GPT-4's capabilities on medical challenge benchmarks in the absence of special training.
We find that prompting innovation can unlock deeper specialist capabilities and show that GPT-4 easily tops prior leading results for medical benchmarks.
With Medprompt, GPT-4 achieves state-of-the-art results on all nine of the benchmark datasets in the MultiMedQA suite.
arXiv Detail & Related papers (2023-11-28T03:16:12Z) - ProBio: A Protocol-guided Multimodal Dataset for Molecular Biology Lab [67.24684071577211]
The challenge of replicating research results has posed a significant impediment to the field of molecular biology.
We first curate a comprehensive multimodal dataset, named ProBio, as an initial step towards this objective.
Next, we devise two challenging benchmarks, transparent solution tracking and multimodal action recognition, to emphasize the unique characteristics and difficulties associated with activity understanding in BioLab settings.
arXiv Detail & Related papers (2023-11-01T14:44:01Z) - Knowledge-injected Prompt Learning for Chinese Biomedical Entity
Normalization [6.927883826415262]
We propose a novel Knowledge-injected Prompt Learning (PL-Knowledge) method to tackle the Biomedical Entity Normalization (BEN) task.
Specifically, our approach consists of five stages: candidate entity matching, knowledge extraction, knowledge encoding, knowledge injection, and prediction output.
By effectively encoding the knowledge items contained in medical entities, the additional knowledge enhances the model's ability to capture latent relationships between medical entities.
arXiv Detail & Related papers (2023-08-23T09:32:40Z) - Self-Verification Improves Few-Shot Clinical Information Extraction [73.6905567014859]
Large language models (LLMs) have shown the potential to accelerate clinical curation via few-shot in-context learning.
They still struggle with issues regarding accuracy and interpretability, especially in mission-critical domains such as health.
Here, we explore a general mitigation framework using self-verification, which leverages the LLM to provide provenance for its own extraction and check its own outputs.
arXiv Detail & Related papers (2023-05-30T22:05:11Z) - Injecting Knowledge into Biomedical Pre-trained Models via Polymorphism
and Synonymous Substitution [22.471123408160658]
Pre-trained language models (PLMs) were considered to be able to store relational knowledge present in the training data.
Low-frequency relational knowledge might be underexpressed compared to high-frequency one in PLMs.
We propose a simple-yet-effective approach to inject relational knowledge into PLMs.
arXiv Detail & Related papers (2023-05-24T10:48:53Z) - Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus.
We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
arXiv Detail & Related papers (2023-05-15T15:47:09Z) - Context Variance Evaluation of Pretrained Language Models for
Prompt-based Biomedical Knowledge Probing [9.138354194112395]
We show that prompt-based probing methods can only probe a lower bound of knowledge.
We introduce context variance into the prompt generation and propose a new rank-change-based evaluation metric.
arXiv Detail & Related papers (2022-11-18T14:44:09Z) - Self-Supervised Knowledge Assimilation for Expert-Layman Text Style
Transfer [63.72621204057025]
Expert-layman text style transfer technologies have the potential to improve communication between scientific communities and the general public.
High-quality information produced by experts is often filled with difficult jargon laypeople struggle to understand.
This is a particularly notable issue in the medical domain, where layman are often confused by medical text online.
arXiv Detail & Related papers (2021-10-06T17:57:22Z) - Can Language Models be Biomedical Knowledge Bases? [18.28724653601921]
We create the BioLAMA benchmark comprised of 49K biomedical factual knowledge triples for probing biomedical LMs.
We find that biomedical LMs with recently proposed probing methods can achieve up to 18.51% Acc@5 on retrieving biomedical knowledge.
arXiv Detail & Related papers (2021-09-15T08:34:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.