Measure More, Question More: Experimental Studies on Transformer-based Language Models and Complement Coercion

• URL: http://arxiv.org/abs/2212.10536v1
• Date: Tue, 20 Dec 2022 18:46:20 GMT
• Title: Measure More, Question More: Experimental Studies on Transformer-based Language Models and Complement Coercion
• Abstract summary: Transformer-based language models have shown strong performance on an array of natural language understanding tasks. We investigate how these models react to implicit meaning using the complement coercion phenomenon.
• Score: 2.1320960069210484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based language models have shown strong performance on an array of natural language understanding tasks. However, the question of how these models react to implicit meaning has been largely unexplored. We investigate this using the complement coercion phenomenon, which involves sentences like "The student finished the book about sailing" where the action "reading" is implicit. We compare LMs' surprisal estimates at various critical sentence regions in sentences with and without implicit meaning. Effects associated with recovering implicit meaning were found at a critical region other than where sentences minimally differ. We then use follow-up experiments to factor out potential confounds, revealing different perspectives that offer a richer and more accurate picture.

Related papers

• Trajectory Dynamics in Language Model Hidden States Predict Human Processing Costs Beyond Surprisal [0.0]
  We show that word-level prediction error (surprisal) and sensitivity to the local momentum of the unfolding interpretation (trajectory extrapolation error) are dissociable components of processing cost.<n>The effect is especially pronounced in garden-path sentences.
  arXiv  Detail & Related papers  (2026-06-03T18:44:41Z)
• Floating or Suggesting Ideas? A Large-Scale Contrastive Analysis of Metaphorical and Literal Verb-Object Constructions [53.690096725532726]
  We analyze 297 English verb-object pairs (e.g., float idea vs. suggest idea) in 2M corpus sentences.<n>Using five NLP tools, we extract 2,293 cognitive and linguistic features capturing affective, lexical, syntactic, and discourse-level properties.<n>Cross-pair results show literal contexts have higher lexical frequency, cohesion, and structural regularity, while metaphorical contexts show greater affective load, imageability, lexical diversity, and constructional specificity.
  arXiv  Detail & Related papers  (2026-04-09T14:08:57Z)
• Surprisal and Metaphor Novelty: Moderate Correlations and Divergent Scaling Effects [9.85813493536013]
  This study investigates whether surprisal, a probabilistic measure of predictability in language models, correlates with metaphor novelty datasets.<n>We analyse surprisal from 16 LM variants on corpus-based and synthetic metaphor novelty datasets.
  arXiv  Detail & Related papers  (2026-01-05T11:24:33Z)
• Counterfactual reasoning: an analysis of in-context emergence [49.58529868457226]
  Large-scale neural language models (LMs) exhibit remarkable performance in in-context learning.<n>This work studies in-context counterfactual reasoning in language models, that is, to predict the consequences of changes under hypothetical scenarios.
  arXiv  Detail & Related papers  (2025-06-05T16:02:07Z)
• Truth-value judgment in language models: belief directions are context sensitive [2.324913904215885]
  latent spaces of large language models contain directions predictive of the truth of sentences. Multiple methods recover such directions and build probes that are described as getting at a model's "knowledge" or "beliefs" We investigate this phenomenon, looking closely at the impact of context on the probes.
  arXiv  Detail & Related papers  (2024-04-29T16:52:57Z)
• UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations [62.71847873326847]
  We investigate the ability to model unusual, unexpected, and unlikely situations. Given a piece of context with an unexpected outcome, this task requires reasoning abductively to generate an explanation. We release a new English language corpus called UNcommonsense.
  arXiv  Detail & Related papers  (2023-11-14T19:00:55Z)
• Prompting Large Language Model for Machine Translation: A Case Study [87.88120385000666]
  We offer a systematic study on prompting strategies for machine translation. We examine factors for prompt template and demonstration example selection. We explore the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning.
  arXiv  Detail & Related papers  (2023-01-17T18:32:06Z)
• Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Models [91.3755431537592]
  Representing compositional and non-compositional phrases is critical for language understanding. We first formulate a problem of predicting the LM-internal representations of longer phrases given those of their constituents. While we would expect the predictive accuracy to correlate with human judgments of semantic compositionality, we find this is largely not the case.
  arXiv  Detail & Related papers  (2022-10-07T14:21:30Z)
• Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
  We suggest a naturalistic strategy for input-level intervention on real world data in Spanish. Using our approach, we isolate morpho-syntactic features from counfounders in sentences. We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
  arXiv  Detail & Related papers  (2022-05-14T11:47:58Z)
• Human Interpretation of Saliency-based Explanation Over Text [65.29015910991261]
  We study saliency-based explanations over textual data. We find that people often mis-interpret the explanations. We propose a method to adjust saliencies based on model estimates of over- and under-perception.
  arXiv  Detail & Related papers  (2022-01-27T15:20:32Z)
• Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge [59.22170796793179]
  Transformers Language Models (TLMs) were tested on a benchmark for the textitdynamic estimation of thematic fit Our results show that TLMs can reach performances that are comparable to those achieved by SDM. However, additional analysis consistently suggests that TLMs do not capture important aspects of event knowledge.
  arXiv  Detail & Related papers  (2021-07-22T20:52:26Z)
• Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy [0.0]
  We assess the ability of both static and contextualized models to adequately represent different lexical-semantic relations. Experiments are performed in Galician, Portuguese, English, and Spanish.
  arXiv  Detail & Related papers  (2021-06-25T10:54:23Z)
• On the Interplay Between Fine-tuning and Composition in Transformers [7.513100214864645]
  We investigate the impact of fine-tuning on the capacity of contextualized embeddings to capture phrase meaning information. Specifically, we fine-tune models on an adversarial paraphrase classification task with high lexical overlap, and on a sentiment classification task. We find that fine-tuning largely fails to benefit compositionality in these representations, though training on sentiment yields a small, localized benefit for certain models.
  arXiv  Detail & Related papers  (2021-05-31T01:49:56Z)
• Narrative Incoherence Detection [76.43894977558811]
  We propose the task of narrative incoherence detection as a new arena for inter-sentential semantic understanding. Given a multi-sentence narrative, decide whether there exist any semantic discrepancies in the narrative flow.
  arXiv  Detail & Related papers  (2020-12-21T07:18:08Z)
• Characterizing the Effect of Sentence Context on Word Meanings: Mapping Brain to Behavior [0.0]
  This paper aims to answer whether the subjects are aware of such changes and agree with them. Subjects were asked to judge how the word change from their generic meaning when the words were used in specific sentences. Results support the hypothesis that word meaning change systematically depending on sentence context.
  arXiv  Detail & Related papers  (2020-07-27T20:12:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.