Decomposition of surprisal: Unified computational model of ERP components in language processing

• URL: http://arxiv.org/abs/2409.06803v1
• Date: Tue, 10 Sep 2024 18:14:02 GMT
• Title: Decomposition of surprisal: Unified computational model of ERP components in language processing
• Authors: Jiaxuan Li, Richard Futrell,
• Abstract summary: We advance an information-theoretic model of human language processing in the brain in which incoming linguistic input is processed at first shallowly and later with more depth. We show that the information content (surprisal) of a word in context can be decomposed into two quantities: (A) surprise, which signals shallow processing difficulty for a word, and (B) discrepancy signal, which reflects the discrepancy between shallow and deep interpretations.
• Score: 7.760815504640362
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The functional interpretation of language-related ERP components has been a central debate in psycholinguistics for decades. We advance an information-theoretic model of human language processing in the brain in which incoming linguistic input is processed at first shallowly and later with more depth, with these two kinds of information processing corresponding to distinct electroencephalographic signatures. Formally, we show that the information content (surprisal) of a word in context can be decomposed into two quantities: (A) heuristic surprise, which signals shallow processing difficulty for a word, and corresponds with the N400 signal; and (B) discrepancy signal, which reflects the discrepancy between shallow and deep interpretations, and corresponds to the P600 signal. Both of these quantities can be estimated straightforwardly using modern NLP models. We validate our theory by successfully simulating ERP patterns elicited by a variety of linguistic manipulations in previously-reported experimental data from six experiments, with successful novel qualitative and quantitative predictions. Our theory is compatible with traditional cognitive theories assuming a `good-enough' heuristic interpretation stage, but with a precise information-theoretic formulation. The model provides an information-theoretic model of ERP components grounded on cognitive processes, and brings us closer to a fully-specified neuro-computational model of language processing.

Related papers

• Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network [16.317199232071232]
  Large Language Models (LLMs) have been shown to be effective models of the human language system. In this work, we investigate the key architectural components driving the surprising alignment of untrained models.
  arXiv  Detail & Related papers  (2024-06-21T12:54:03Z)
• An information-theoretic model of shallow and deep language comprehension [7.760815504640362]
  We use information theory to formulate a model of language comprehension as an optimal trade-off between accuracy and processing depth. By quantifying the timecourse of language processing as it proceeds from shallow to deep, our model provides a unified framework to explain behavioral and neural signatures of language comprehension.
  arXiv  Detail & Related papers  (2024-05-13T22:29:33Z)
• Explaining Text Similarity in Transformer Models [52.571158418102584]
  Recent advances in explainable AI have made it possible to mitigate limitations by leveraging improved explanations for Transformers. We use BiLRP, an extension developed for computing second-order explanations in bilinear similarity models, to investigate which feature interactions drive similarity in NLP models. Our findings contribute to a deeper understanding of different semantic similarity tasks and models, highlighting how novel explainable AI methods enable in-depth analyses and corpus-level insights.
  arXiv  Detail & Related papers  (2024-05-10T17:11:31Z)
• A unified information-theoretic model of EEG signatures of human language processing [7.190747604294439]
  We advance an information-theoretic model of human language processing in the brain, in which incoming linguistic input is processed at two levels. We propose that these two kinds of information processing have distinct electroencephalographic signatures, corresponding to the well-documented N400 and P600 components of language-related event-related potentials (ERPs) Our theory is in principle compatible with traditional cognitive theories assuming a good-enough' interpretation stage, but with precise information-theoretic formulation.
  arXiv  Detail & Related papers  (2022-12-16T00:15:45Z)
• A Unified Understanding of Deep NLP Models for Text Classification [88.35418976241057]
  We have developed a visual analysis tool, DeepNLPVis, to enable a unified understanding of NLP models for text classification. The key idea is a mutual information-based measure, which provides quantitative explanations on how each layer of a model maintains the information of input words in a sample. A multi-level visualization, which consists of a corpus-level, a sample-level, and a word-level visualization, supports the analysis from the overall training set to individual samples.
  arXiv  Detail & Related papers  (2022-06-19T08:55:07Z)
• Dependency-based Mixture Language Models [53.152011258252315]
  We introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention.
  arXiv  Detail & Related papers  (2022-03-19T06:28:30Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Does injecting linguistic structure into language models lead to better alignment with brain recordings? [13.880819301385854]
  We evaluate whether language models align better with brain recordings if their attention is biased by annotations from syntactic or semantic formalisms. Our proposed approach enables the evaluation of more targeted hypotheses about the composition of meaning in the brain.
  arXiv  Detail & Related papers  (2021-01-29T14:42:02Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)
• Data Augmentation for Spoken Language Understanding via Pretrained Language Models [113.56329266325902]
  Training of spoken language understanding (SLU) models often faces the problem of data scarcity. We put forward a data augmentation method using pretrained language models to boost the variability and accuracy of generated utterances.
  arXiv  Detail & Related papers  (2020-04-29T04:07:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.