Integrating large language models and active inference to understand eye movements in reading and dyslexia

• URL: http://arxiv.org/abs/2308.04941v2
• Date: Sat, 24 Feb 2024 10:55:22 GMT
• Title: Integrating large language models and active inference to understand eye movements in reading and dyslexia
• Authors: Francesco Donnarumma, Mirco Frosolone and Giovanni Pezzulo
• Abstract summary: We present a novel computational model employing hierarchical active inference to simulate reading and eye movements. Our model permits the exploration of maladaptive inference effects on eye movements during reading, such as in dyslexia.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present a novel computational model employing hierarchical active inference to simulate reading and eye movements. The model characterizes linguistic processing as inference over a hierarchical generative model, facilitating predictions and inferences at various levels of granularity, from syllables to sentences. Our approach combines the strengths of large language models for realistic textual predictions and active inference for guiding eye movements to informative textual information, enabling the testing of predictions. The model exhibits proficiency in reading both known and unknown words and sentences, adhering to the distinction between lexical and nonlexical routes in dual-route theories of reading. Notably, our model permits the exploration of maladaptive inference effects on eye movements during reading, such as in dyslexia. To simulate this condition, we attenuate the contribution of priors during the reading process, leading to incorrect inferences and a more fragmented reading style, characterized by a greater number of shorter saccades. This alignment with empirical findings regarding eye movements in dyslexic individuals highlights the model's potential to aid in understanding the cognitive processes underlying reading and eye movements, as well as how reading deficits associated with dyslexia may emerge from maladaptive predictive processing. In summary, our model represents a significant advancement in comprehending the intricate cognitive processes involved in reading and eye movements, with potential implications for understanding and addressing dyslexia through the simulation of maladaptive inference. It may offer valuable insights into this condition and contribute to the development of more effective interventions for treatment.

Related papers

• A generative framework to bridge data-driven models and scientific theories in language neuroscience [84.76462599023802]
  We present generative explanation-mediated validation, a framework for generating concise explanations of language selectivity in the brain. We show that explanatory accuracy is closely related to the predictive power and stability of the underlying statistical models.
  arXiv  Detail & Related papers  (2024-10-01T15:57:48Z)
• Tokensome: Towards a Genetic Vision-Language GPT for Explainable and Cognitive Karyotyping [7.101960527853822]
  Tokensome is a novel vision-language model based on chromosome tokenization for explainable and cognitive karyotyping. Tokensome elevates the method from the conventional visual perception layer to the cognitive decision-making layer.
  arXiv  Detail & Related papers  (2024-03-17T03:38:50Z)
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning [67.0609518552321]
  We propose to conduct Machine Vision Therapy which aims to rectify the noisy predictions from vision models. By fine-tuning with the denoised labels, the learning model performance can be boosted in an unsupervised manner.
  arXiv  Detail & Related papers  (2023-12-05T07:29:14Z)
• Zero-Resource Hallucination Prevention for Large Language Models [45.4155729393135]
  "Hallucination" refers to instances where large language models (LLMs) generate factually inaccurate or ungrounded information. We introduce a novel pre-language self-evaluation technique, referred to as SELF-FAMILIARITY, which focuses on evaluating the model's familiarity with the concepts present in the input instruction. We validate SELF-FAMILIARITY across four different large language models, demonstrating consistently superior performance compared to existing techniques.
  arXiv  Detail & Related papers  (2023-09-06T01:57:36Z)
• Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading [3.9766585251585282]
  We develop Eyettention, the first dual-sequence model that simultaneously processes the sequence of words and the chronological sequence of fixations. We show that Eyettention outperforms state-of-the-art models in predicting scanpaths.
  arXiv  Detail & Related papers  (2023-04-21T07:26:49Z)
• Eye-tracking based classification of Mandarin Chinese readers with and without dyslexia using neural sequence models [7.639036130018945]
  We propose two simple sequence models that process eye movements on the entire stimulus without the need of aggregating features across the sentence. We incorporate the linguistic stimulus into the model in two ways -- contextualized word embeddings and manually extracted linguistic features. Our results show that (i) even for a logographic script such as Chinese, sequence models are able to classify dyslexia on eye gaze sequences, reaching state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-10-18T12:57:30Z)
• Causal Reasoning Meets Visual Representation Learning: A Prospective Study [117.08431221482638]
  Lack of interpretability, robustness, and out-of-distribution generalization are becoming the challenges of the existing visual models. Inspired by the strong inference ability of human-level agents, recent years have witnessed great effort in developing causal reasoning paradigms. This paper aims to provide a comprehensive overview of this emerging field, attract attention, encourage discussions, bring to the forefront the urgency of developing novel causal reasoning methods.
  arXiv  Detail & Related papers  (2022-04-26T02:22:28Z)
• Hybrid Predictive Coding: Inferring, Fast and Slow [62.997667081978825]
  We propose a hybrid predictive coding network that combines both iterative and amortized inference in a principled manner. We demonstrate that our model is inherently sensitive to its uncertainty and adaptively balances balances to obtain accurate beliefs using minimum computational expense.
  arXiv  Detail & Related papers  (2022-04-05T12:52:45Z)
• Going Beyond Saliency Maps: Training Deep Models to Interpret Deep Models [16.218680291606628]
  Interpretability is a critical factor in applying complex deep learning models to advance the understanding of brain disorders. We propose to train simulator networks that can warp a given image to inject or remove patterns of the disease. We apply our approach to interpreting classifiers trained on a synthetic dataset and two neuroimaging datasets to visualize the effect of the Alzheimer's disease and alcohol use disorder.
  arXiv  Detail & Related papers  (2021-02-16T15:57:37Z)
• On-the-Fly Attention Modularization for Neural Generation [54.912042110885366]
  We show that generated text is repetitive, generic, self-inconsistent, and lacking commonsense. Our findings motivate on-the-fly attention modularization, a simple but effective method for injecting inductive biases into attention during inference.
  arXiv  Detail & Related papers  (2021-01-02T05:16:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.