A Matter of Framing: The Impact of Linguistic Formalism on Probing Results

• URL: http://arxiv.org/abs/2004.14999v1
• Date: Thu, 30 Apr 2020 17:45:16 GMT
• Title: A Matter of Framing: The Impact of Linguistic Formalism on Probing Results
• Authors: Ilia Kuznetsov, Iryna Gurevych
• Abstract summary: Deep pre-trained contextualized encoders like BERT (Delvin et al.) demonstrate remarkable performance on a range of downstream tasks. Recent research in probing investigates the linguistic knowledge implicitly learned by these models during pre-training. Can the choice of formalism affect probing results? We find linguistically meaningful differences in the encoding of semantic role- and proto-role information by BERT depending on the formalism.
• Score: 69.36678873492373
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep pre-trained contextualized encoders like BERT (Delvin et al., 2019) demonstrate remarkable performance on a range of downstream tasks. A recent line of research in probing investigates the linguistic knowledge implicitly learned by these models during pre-training. While most work in probing operates on the task level, linguistic tasks are rarely uniform and can be represented in a variety of formalisms. Any linguistics-based probing study thereby inevitably commits to the formalism used to annotate the underlying data. Can the choice of formalism affect probing results? To investigate, we conduct an in-depth cross-formalism layer probing study in role semantics. We find linguistically meaningful differences in the encoding of semantic role- and proto-role information by BERT depending on the formalism and demonstrate that layer probing can detect subtle differences between the implementations of the same linguistic formalism. Our results suggest that linguistic formalism is an important dimension in probing studies, along with the commonly used cross-task and cross-lingual experimental settings.

Related papers

• What does BERT learn about prosody? [1.1548853370822343]
  We study whether prosody is part of the structural information of the language that models learn. Our results show that information about prosodic prominence spans across many layers but is mostly focused in middle layers suggesting that BERT relies mostly on syntactic and semantic information.
  arXiv  Detail & Related papers  (2023-04-25T10:34:56Z)
• Transfer Learning of Lexical Semantic Families for Argumentative Discourse Units Identification [0.8508198765617198]
  Argument mining tasks require an informed range of low to high complexity linguistic phenomena and commonsense knowledge. Previous work has shown that pre-trained language models are highly effective at encoding syntactic and semantic linguistic phenomena. It remains an issue of how much the existing pre-trained language models encompass the complexity of argument mining tasks.
  arXiv  Detail & Related papers  (2022-09-06T13:38:47Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Probing Linguistic Information For Logical Inference In Pre-trained Language Models [2.4366811507669124]
  We propose a methodology for probing linguistic information for logical inference in pre-trained language model representations. We find that (i) pre-trained language models do encode several types of linguistic information for inference, but there are also some types of information that are weakly encoded. We have demonstrated language models' potential as semantic and background knowledge bases for supporting symbolic inference methods.
  arXiv  Detail & Related papers  (2021-12-03T07:19:42Z)
• Probing Pretrained Language Models for Lexical Semantics [76.73599166020307]
  We present a systematic empirical analysis across six typologically diverse languages and five different lexical tasks. Our results indicate patterns and best practices that hold universally, but also point to prominent variations across languages and tasks.
  arXiv  Detail & Related papers  (2020-10-12T14:24:01Z)
• Intrinsic Probing through Dimension Selection [69.52439198455438]
  Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks. Such high performance should not be possible unless some form of linguistic structure inheres in these representations, and a wealth of research has sprung up on probing for it. In this paper, we draw a distinction between intrinsic probing, which examines how linguistic information is structured within a representation, and the extrinsic probing popular in prior work, which only argues for the presence of such information by showing that it can be successfully extracted.
  arXiv  Detail & Related papers  (2020-10-06T15:21:08Z)
• Information-Theoretic Probing for Linguistic Structure [74.04862204427944]
  We propose an information-theoretic operationalization of probing as estimating mutual information. We evaluate on a set of ten typologically diverse languages often underrepresented in NLP research.
  arXiv  Detail & Related papers  (2020-04-07T01:06:36Z)
• On the Importance of Word Order Information in Cross-lingual Sequence Labeling [80.65425412067464]
  Cross-lingual models that fit into the word order of the source language might fail to handle target languages. We investigate whether making models insensitive to the word order of the source language can improve the adaptation performance in target languages.
  arXiv  Detail & Related papers  (2020-01-30T03:35:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.