Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language

• URL: http://arxiv.org/abs/2201.10262v1
• Date: Tue, 25 Jan 2022 12:11:46 GMT
• Title: Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language
• Authors: Mael Jullien, Marco Valentino, Andre Freitas
• Abstract summary: We present a systematic Foundational Ontology probing methodology to investigate whether Transformers-based models encode abstract semantic information. We show that Transformer-based models incidentally encode information related to Foundational Ontologies during the pre-training pro-cess.
• Score: 2.363388546004777
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the methodological support of probing (or diagnostic classification), recent studies have demonstrated that Transformers encode syntactic and semantic information to some extent. Following this line of research, this paper aims at taking semantic probing to an abstraction extreme with the goal of answering the following research question: can contemporary Transformer-based models reflect an underlying Foundational Ontology? To this end, we present a systematic Foundational Ontology (FO) probing methodology to investigate whether Transformers-based models encode abstract semantic information. Following different pre-training and fine-tuning regimes, we present an extensive evaluation of a diverse set of large-scale language models over three distinct and complementary FO tagging experiments. Specifically, we present and discuss the following conclusions: (1) The probing results indicate that Transformer-based models incidentally encode information related to Foundational Ontologies during the pre-training pro-cess; (2) Robust FO taggers (accuracy of 90 percent)can be efficiently built leveraging on this knowledge.

Related papers

• A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis [0.8049701904919515]
  This paper introduces the major developments of Transformer-based models in the recent past in the context of nucleotide sequences. We believe this review will help the scientific community in understanding the various applications of Transformer-based language models to nucleotide sequences.
  arXiv  Detail & Related papers  (2024-12-10T05:33:09Z)
• Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis [63.66763657191476]
  We show that efficient numerical training and inference algorithms as low-rank computation have impressive performance for learning Transformer-based adaption. We analyze how magnitude-based models affect generalization while improving adaption. We conclude that proper magnitude-based has a slight on the testing performance.
  arXiv  Detail & Related papers  (2024-06-24T23:00:58Z)
• Hidden Holes: topological aspects of language models [1.1172147007388977]
  We study the evolution of topological structure in GPT based large language models across depth and time during training. We show that the latter exhibit more topological complexity, with a distinct pattern of changes common to all natural languages but absent from synthetically generated data.
  arXiv  Detail & Related papers  (2024-06-09T14:25:09Z)
• Explaining Text Similarity in Transformer Models [52.571158418102584]
  Recent advances in explainable AI have made it possible to mitigate limitations by leveraging improved explanations for Transformers. We use BiLRP, an extension developed for computing second-order explanations in bilinear similarity models, to investigate which feature interactions drive similarity in NLP models. Our findings contribute to a deeper understanding of different semantic similarity tasks and models, highlighting how novel explainable AI methods enable in-depth analyses and corpus-level insights.
  arXiv  Detail & Related papers  (2024-05-10T17:11:31Z)
• UMLS-KGI-BERT: Data-Centric Knowledge Integration in Transformers for Biomedical Entity Recognition [4.865221751784403]
  This work contributes a data-centric paradigm for enriching the language representations of biomedical transformer-encoder LMs by extracting text sequences from the UMLS. Preliminary results from experiments in the extension of pre-trained LMs as well as training from scratch show that this framework improves downstream performance on multiple biomedical and clinical Named Entity Recognition (NER) tasks.
  arXiv  Detail & Related papers  (2023-07-20T18:08:34Z)
• All Roads Lead to Rome? Exploring the Invariance of Transformers' Representations [69.3461199976959]
  We propose a model based on invertible neural networks, BERT-INN, to learn the Bijection Hypothesis. We show the advantage of BERT-INN both theoretically and through extensive experiments.
  arXiv  Detail & Related papers  (2023-05-23T22:30:43Z)
• A Closer Look at Few-shot Classification Again [68.44963578735877]
  Few-shot classification consists of a training phase and an adaptation phase. We empirically prove that the training algorithm and the adaptation algorithm can be completely disentangled. Our meta-analysis for each phase reveals several interesting insights that may help better understand key aspects of few-shot classification.
  arXiv  Detail & Related papers  (2023-01-28T16:42:05Z)
• Application of Deep Learning in Generating Structured Radiology Reports: A Transformer-Based Technique [0.4549831511476247]
  Natural language processing techniques can facilitate automatic information extraction and transformation of free-text formats to structured data. Deep learning (DL)-based models have been adapted for NLP experiments with promising results. In this study, we propose a transformer-based fine-grained named entity recognition architecture for clinical information extraction.
  arXiv  Detail & Related papers  (2022-09-25T08:03:15Z)
• Analyzing Transformers in Embedding Space [59.434807802802105]
  We present a theoretical analysis where all parameters of a trained Transformer are interpreted by projecting them into the embedding space. We show that parameters of both pretrained and fine-tuned models can be interpreted in embedding space. Our findings open the door to interpretation methods that, at least in part, abstract away from model specifics and operate in the embedding space only.
  arXiv  Detail & Related papers  (2022-09-06T14:36:57Z)
• Unnatural Language Inference [48.45003475966808]
  We find that state-of-the-art NLI models, such as RoBERTa and BART, are invariant to, and sometimes even perform better on, examples with randomly reordered words. Our findings call into question the idea that our natural language understanding models, and the tasks used for measuring their progress, genuinely require a human-like understanding of syntax.
  arXiv  Detail & Related papers  (2020-12-30T20:40:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.