Testing the limits of natural language models for predicting human language judgments

• URL: http://arxiv.org/abs/2204.03592v3
• Date: Tue, 12 Sep 2023 15:01:52 GMT
• Title: Testing the limits of natural language models for predicting human language judgments
• Abstract summary: We compared the model-human consistency of diverse language models using controversial sentence pairs. Controversial sentence pairs proved highly effective at revealing model failures. The most human-consistent model tested was GPT-2, although experiments also revealed significant shortcomings of its alignment with human perception.
• Score: 0.27309692684728604
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural network language models can serve as computational hypotheses about how humans process language. We compared the model-human consistency of diverse language models using a novel experimental approach: controversial sentence pairs. For each controversial sentence pair, two language models disagree about which sentence is more likely to occur in natural text. Considering nine language models (including n-gram, recurrent neural networks, and transformer models), we created hundreds of such controversial sentence pairs by either selecting sentences from a corpus or synthetically optimizing sentence pairs to be highly controversial. Human subjects then provided judgments indicating for each pair which of the two sentences is more likely. Controversial sentence pairs proved highly effective at revealing model failures and identifying models that aligned most closely with human judgments. The most human-consistent model tested was GPT-2, although experiments also revealed significant shortcomings of its alignment with human perception.

Related papers

• How Unlikely Is "Unlikely"? Assessing Verbal Probability Perception Across Large Language Models [1.435579512127234]
  We present a systematic cross-model evaluation using a word-to-number mapping task grounded in established human benchmarks.<n>Eleven uncertainty expressions were presented to 19 models under two conditions, forced single-number response and explanation elicitation.<n>Models show a systematic upward bias for negative expressions such as unlikely'' and improbable''
  arXiv  Detail & Related papers  (2026-08-26T19:00:41Z)
• When transformers learn "impossible" languages, what do they learn? [9.276738475766512]
  We evaluate two theoretically motivated linking hypotheses: impossibility arising from deficiencies in grammatical sensitivity or generative production.<n>Using GPT-2 style models trained on perturbed "impossible" variants of English, we measure sensitivity to grammaticality.<n>We find that model performance exhibits only gradual degradation, mediated by the language's information locality.
  arXiv  Detail & Related papers  (2026-06-29T18:42:03Z)
• A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles [0.06554326244334868]
  We evaluate large language models' sensitivity to argument roles by replicating psycholinguistic studies on human argument role processing. We find that language models are able to distinguish verbs that appear in plausible and implausible contexts, where plausibility is determined through the relation between the verb and its preceding arguments. This indicates that language models' capacity to detect verb plausibility does not arise from the same mechanism that underlies human real-time sentence processing.
  arXiv  Detail & Related papers  (2024-10-21T16:05:58Z)
• A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors [50.046717886067555]
  We show that when sampling corpora from an aligned language model, there exists a trade-off between the strings' average reward and average log-likelihood. We provide a formal treatment of this phenomenon and demonstrate how a choice of sampling adaptor allows for a selection of how much likelihood we exchange for the reward.
  arXiv  Detail & Related papers  (2024-06-14T17:38:21Z)
• SpeechAlign: Aligning Speech Generation to Human Preferences [51.684183257809075]
  We introduce SpeechAlign, an iterative self-improvement strategy that aligns speech language models to human preferences. We show that SpeechAlign can bridge the distribution gap and facilitate continuous self-improvement of the speech language model.
  arXiv  Detail & Related papers  (2024-04-08T15:21:17Z)
• UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations [62.71847873326847]
  We investigate the ability to model unusual, unexpected, and unlikely situations. Given a piece of context with an unexpected outcome, this task requires reasoning abductively to generate an explanation. We release a new English language corpus called UNcommonsense.
  arXiv  Detail & Related papers  (2023-11-14T19:00:55Z)
• Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifiers [0.6091702876917281]
  We focus on 'few'-type quantifiers, as in 'few children like toys', which might pose a particular challenge for language models. We present 960 English sentence stimuli from two human neurolinguistic experiments to 22 autoregressive transformer models of differing sizes.
  arXiv  Detail & Related papers  (2022-12-16T20:01:22Z)
• A fine-grained comparison of pragmatic language understanding in humans and language models [2.231167375820083]
  We compare language models and humans on seven pragmatic phenomena. We find that the largest models achieve high accuracy and match human error patterns. Preliminary evidence that models and humans are sensitive to similar linguistic cues.
  arXiv  Detail & Related papers  (2022-12-13T18:34:59Z)
• Can language models handle recursively nested grammatical structures? A case study on comparing models and humans [0.0]
  How should we compare the capabilities of language models and humans? I consider a case study: processing of nested grammatical structures. I suggest that there is an important difference between evaluating cognitive models of a specific phenomenon and evaluating broadly-trained models.
  arXiv  Detail & Related papers  (2022-10-27T10:25:12Z)
• Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
  We suggest a naturalistic strategy for input-level intervention on real world data in Spanish. Using our approach, we isolate morpho-syntactic features from counfounders in sentences. We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
  arXiv  Detail & Related papers  (2022-05-14T11:47:58Z)
• Do Multilingual Language Models Capture Differing Moral Norms? [71.52261949766101]
  Massively multilingual sentence representations are trained on large corpora of uncurated data. This may cause the models to grasp cultural values including moral judgments from the high-resource languages. The lack of data in certain languages can also lead to developing random and thus potentially harmful beliefs.
  arXiv  Detail & Related papers  (2022-03-18T12:26:37Z)
• A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans [2.7624021966289605]
  We present a scaled-up comparison of incremental processing in humans and neural language models. Data comes from a novel online experimental paradigm called the Interpolated Maze task. We find that both humans and language models show increased processing difficulty in ungrammatical sentence regions.
  arXiv  Detail & Related papers  (2021-06-06T20:04:39Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)
• Limits of Detecting Text Generated by Large-Scale Language Models [65.46403462928319]
  Some consider large-scale language models that can generate long and coherent pieces of text as dangerous, since they may be used in misinformation campaigns. Here we formulate large-scale language model output detection as a hypothesis testing problem to classify text as genuine or generated.
  arXiv  Detail & Related papers  (2020-02-09T19:53:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.