Multilingual Language Models Predict Human Reading Behavior

• URL: http://arxiv.org/abs/2104.05433v1
• Date: Mon, 12 Apr 2021 13:03:49 GMT
• Title: Multilingual Language Models Predict Human Reading Behavior
• Authors: Nora Hollenstein, Federico Pirovano, Ce Zhang, Lena J\"ager and Lisa Beinborn
• Abstract summary: We compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures. We find that BERT and XLM models successfully predict a range of eye tracking features. In a series of experiments, we analyze the cross-domain and cross-language abilities of these models and show how they reflect human sentence processing.
• Score: 8.830621849672108
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We analyze if large language models are able to predict patterns of human reading behavior. We compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures reflecting natural human sentence processing on Dutch, English, German, and Russian texts. This results in accurate models of human reading behavior, which indicates that transformer models implicitly encode relative importance in language in a way that is comparable to human processing mechanisms. We find that BERT and XLM models successfully predict a range of eye tracking features. In a series of experiments, we analyze the cross-domain and cross-language abilities of these models and show how they reflect human sentence processing.

Related papers

• Reverse-Engineering the Reader [43.26660964074272]
  We introduce a novel alignment technique in which we fine-tune a language model to implicitly optimize the parameters of a linear regressor. Using words as a test case, we evaluate our technique across multiple model sizes and datasets. We find an inverse relationship between psychometric power and a model's performance on downstream NLP tasks as well as its perplexity on held-out test data.
  arXiv  Detail & Related papers  (2024-10-16T23:05:01Z)
• DevBench: A multimodal developmental benchmark for language learning [0.34129029452670606]
  We introduce DevBench, a benchmark for evaluating vision-language models on tasks and behavioral data. We show that DevBench provides a benchmark for comparing models to human language development. These comparisons highlight ways in which model and human language learning processes diverge.
  arXiv  Detail & Related papers  (2024-06-14T17:49:41Z)
• Humans and language models diverge when predicting repeating text [52.03471802608112]
  We present a scenario in which the performance of humans and LMs diverges. Human and GPT-2 LM predictions are strongly aligned in the first presentation of a text span, but their performance quickly diverges when memory begins to play a role. We hope that this scenario will spur future work in bringing LMs closer to human behavior.
  arXiv  Detail & Related papers  (2023-10-10T08:24:28Z)
• Can Language Models Learn to Listen? [96.01685069483025]
  We present a framework for generating appropriate facial responses from a listener in dyadic social interactions based on the speaker's words. Our approach autoregressively predicts a response of a listener: a sequence of listener facial gestures, quantized using a VQ-VAE. We show that our generated listener motion is fluent and reflective of language semantics through quantitative metrics and a qualitative user study.
  arXiv  Detail & Related papers  (2023-08-21T17:59:02Z)
• Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? [9.909170013118775]
  The propensity of larger Transformer-based models to'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations. These results suggest that the propensity of larger Transformer-based models to'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations.
  arXiv  Detail & Related papers  (2022-12-23T03:57:54Z)
• What do Large Language Models Learn beyond Language? [10.9650651784511]
  We find that pretrained models significantly outperform comparable non-pretrained neural models. Experiments surprisingly reveal that the positive effects of pre-training persist even when pretraining on multi-lingual text or computer code. Our findings suggest a hitherto unexplored deep connection between pre-training and inductive learning abilities of language models.
  arXiv  Detail & Related papers  (2022-10-21T23:43:13Z)
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher [83.98181046650664]
  We present an analysis of Transformer-based language model performance across a wide range of model scales. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language. We discuss the application of language models to AI safety and the mitigation of downstream harms.
  arXiv  Detail & Related papers  (2021-12-08T19:41:47Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)
• A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans [2.7624021966289605]
  We present a scaled-up comparison of incremental processing in humans and neural language models. Data comes from a novel online experimental paradigm called the Interpolated Maze task. We find that both humans and language models show increased processing difficulty in ungrammatical sentence regions.
  arXiv  Detail & Related papers  (2021-06-06T20:04:39Z)
• Improving Cross-Lingual Reading Comprehension with Self-Training [62.73937175625953]
  Current state-of-the-art models even surpass human performance on several benchmarks. Previous works have revealed the abilities of pre-trained multilingual models for zero-shot cross-lingual reading comprehension. This paper further utilized unlabeled data to improve the performance.
  arXiv  Detail & Related papers  (2021-05-08T08:04:30Z)
• Pre-Training a Language Model Without Human Language [74.11825654535895]
  We study how the intrinsic nature of pre-training data contributes to the fine-tuned downstream performance. We find that models pre-trained on unstructured data beat those trained directly from scratch on downstream tasks. To our great astonishment, we uncover that pre-training on certain non-human language data gives GLUE performance close to performance pre-trained on another non-English language.
  arXiv  Detail & Related papers  (2020-12-22T13:38:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.