Divergences between Language Models and Human Brains

• URL: http://arxiv.org/abs/2311.09308v3
• Date: Mon, 13 Jan 2025 22:22:06 GMT
• Title: Divergences between Language Models and Human Brains
• Authors: Yuchen Zhou, Emmy Liu, Graham Neubig, Michael J. Tarr, Leila Wehbe,
• Abstract summary: We systematically explore the divergences between human and machine language processing.<n>We identify two domains that LMs do not capture well: social/emotional intelligence and physical commonsense.<n>Our results show that fine-tuning LMs on these domains can improve their alignment with human brain responses.
• Score: 59.100552839650774
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Do machines and humans process language in similar ways? Recent research has hinted at the affirmative, showing that human neural activity can be effectively predicted using the internal representations of language models (LMs). Although such results are thought to reflect shared computational principles between LMs and human brains, there are also clear differences in how LMs and humans represent and use language. In this work, we systematically explore the divergences between human and machine language processing by examining the differences between LM representations and human brain responses to language as measured by Magnetoencephalography (MEG) across two datasets in which subjects read and listened to narrative stories. Using an LLM-based data-driven approach, we identify two domains that LMs do not capture well: social/emotional intelligence and physical commonsense. We validate these findings with human behavioral experiments and hypothesize that the gap is due to insufficient representations of social/emotional and physical knowledge in LMs. Our results show that fine-tuning LMs on these domains can improve their alignment with human brain responses.

Related papers

• How Deep is Love in LLMs' Hearts? Exploring Semantic Size in Human-like Cognition [75.11808682808065]
  This study investigates whether large language models (LLMs) exhibit similar tendencies in understanding semantic size. Our findings reveal that multi-modal training is crucial for LLMs to achieve more human-like understanding. Lastly, we examine whether LLMs are influenced by attention-grabbing headlines with larger semantic sizes in a real-world web shopping scenario.
  arXiv  Detail & Related papers  (2025-03-01T03:35:56Z)
• Can Language Models Learn Typologically Implausible Languages? [62.823015163987996]
  Grammatical features across human languages show intriguing correlations often attributed to learning biases in humans. We discuss how language models (LMs) allow us to better determine the role of domain-general learning biases in language universals. We test LMs on an array of highly naturalistic but counterfactual versions of the English (head-initial) and Japanese (head-final) languages.
  arXiv  Detail & Related papers  (2025-02-17T20:40:01Z)
• Human-like conceptual representations emerge from language prediction [72.5875173689788]
  Large language models (LLMs) trained exclusively through next-token prediction over language data exhibit remarkably human-like behaviors. Are these models developing concepts akin to humans, and if so, how are such concepts represented and organized? Our results demonstrate that LLMs can flexibly derive concepts from linguistic descriptions in relation to contextual cues about other concepts. These findings establish that structured, human-like conceptual representations can naturally emerge from language prediction without real-world grounding.
  arXiv  Detail & Related papers  (2025-01-21T23:54:17Z)
• Lost in Translation: The Algorithmic Gap Between LMs and the Brain [8.799971499357499]
  Language Models (LMs) have achieved impressive performance on various linguistic tasks, but their relationship to human language processing in the brain remains unclear. This paper examines the gaps and overlaps between LMs and the brain at different levels of analysis. We discuss how insights from neuroscience, such as sparsity, modularity, internal states, and interactive learning, can inform the development of more biologically plausible language models.
  arXiv  Detail & Related papers  (2024-07-05T17:43:16Z)
• Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network [16.317199232071232]
  Large Language Models (LLMs) have been shown to be effective models of the human language system. In this work, we investigate the key architectural components driving the surprising alignment of untrained models.
  arXiv  Detail & Related papers  (2024-06-21T12:54:03Z)
• Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs [70.3132264719438]
  We aim to fill the research gap by examining how neuron activation is shared across tasks and languages. We classify neurons into four distinct categories based on their responses to a specific input across different languages. Our analysis reveals the following insights: (i) the patterns of neuron sharing are significantly affected by the characteristics of tasks and examples; (ii) neuron sharing does not fully correspond with language similarity; (iii) shared neurons play a vital role in generating responses, especially those shared across all languages.
  arXiv  Detail & Related papers  (2024-06-13T16:04:11Z)
• Do Large Language Models Mirror Cognitive Language Processing? [43.68923267228057]
  Large Language Models (LLMs) have demonstrated remarkable abilities in text comprehension and logical reasoning. In cognitive science, brain cognitive processing signals are typically utilized to study human language processing. We employ Representational Similarity Analysis (RSA) to measure the alignment between 23 mainstream LLMs and fMRI signals of the brain.
  arXiv  Detail & Related papers  (2024-02-28T03:38:20Z)
• Exploring Spatial Schema Intuitions in Large Language and Vision Models [8.944921398608063]
  We investigate whether large language models (LLMs) effectively capture implicit human intuitions about building blocks of language. Surprisingly, correlations between model outputs and human responses emerge, revealing adaptability without a tangible connection to embodied experiences. This research contributes to a nuanced understanding of the interplay between language, spatial experiences, and computations made by large language models.
  arXiv  Detail & Related papers  (2024-02-01T19:25:50Z)
• Instruction-tuning Aligns LLMs to the Human Brain [19.450164922129723]
  We investigate the effect of instruction-tuning on aligning large language models and human language processing mechanisms. We find that instruction-tuning generally enhances brain alignment, but has no similar effect on behavioral alignment. Our results suggest that the mechanisms that encode world knowledge in LLMs also improve representational alignment to the human brain.
  arXiv  Detail & Related papers  (2023-12-01T13:31:02Z)
• The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputs [50.32802502923367]
  We study the process of language driving and influencing social reasoning in a probabilistic goal inference domain. We propose a neuro-symbolic model that carries out goal inference from linguistic inputs of agent scenarios. Our model closely matches human response patterns and better predicts human judgements than using an LLM alone.
  arXiv  Detail & Related papers  (2023-06-25T19:38:01Z)
• Information-Restricted Neural Language Models Reveal Different Brain Regions' Sensitivity to Semantics, Syntax and Context [87.31930367845125]
  We trained a lexical language model, Glove, and a supra-lexical language model, GPT-2, on a text corpus. We then assessed to what extent these information-restricted models were able to predict the time-courses of fMRI signal of humans listening to naturalistic text. Our analyses show that, while most brain regions involved in language are sensitive to both syntactic and semantic variables, the relative magnitudes of these effects vary a lot across these regions.
  arXiv  Detail & Related papers  (2023-02-28T08:16:18Z)
• Toward a realistic model of speech processing in the brain with self-supervised learning [67.7130239674153]
  Self-supervised algorithms trained on the raw waveform constitute a promising candidate. We show that Wav2Vec 2.0 learns brain-like representations with as little as 600 hours of unlabelled speech.
  arXiv  Detail & Related papers  (2022-06-03T17:01:46Z)
• Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects [82.81964713263483]
  A popular approach to decompose the neural bases of language consists in correlating, across individuals, the brain responses to different stimuli. Here, we show that a model-based approach can reach equivalent results within subjects exposed to natural stimuli.
  arXiv  Detail & Related papers  (2021-10-12T15:30:21Z)
• Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses [62.197912623223964]
  We show a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. This suggests that the embedding captures some part of the brain's natural language representation structure.
  arXiv  Detail & Related papers  (2021-06-09T22:59:12Z)
• Presentation and Analysis of a Multimodal Dataset for Grounded Language Learning [32.28310581819443]
  Grounded language acquisition involves learning how language-based interactions refer to the world around them. In practice the data used for learning tends to be cleaner, clearer, and more grammatical than actual human interactions. We present a dataset of common household objects described by people using either spoken or written language.
  arXiv  Detail & Related papers  (2020-07-29T17:58:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.