Towards Automation of Cognitive Modeling using Large Language Models

• URL: http://arxiv.org/abs/2502.00879v1
• Date: Sun, 02 Feb 2025 19:07:13 GMT
• Title: Towards Automation of Cognitive Modeling using Large Language Models
• Authors: Milena Rmus, Akshay K. Jagadish, Marvin Mathony, Tobias Ludwig, Eric Schulz,
• Abstract summary: Computational cognitive models enable researchers to quantify cognitive processes and arbitrate between theories by fitting models to behavioral data. Previous work has demonstrated that Large Language Models (LLMs) are adept at pattern recognition in-context, solving complex problems, and generating executable code. We leverage these abilities to explore the potential of LLMs in automating the generation of cognitive models based on behavioral data.
• Score: 4.269194018613294
• License:
• Abstract: Computational cognitive models, which formalize theories of cognition, enable researchers to quantify cognitive processes and arbitrate between competing theories by fitting models to behavioral data. Traditionally, these models are handcrafted, which requires significant domain knowledge, coding expertise, and time investment. Previous work has demonstrated that Large Language Models (LLMs) are adept at pattern recognition in-context, solving complex problems, and generating executable code. In this work, we leverage these abilities to explore the potential of LLMs in automating the generation of cognitive models based on behavioral data. We evaluated the LLM in two different tasks: model identification (relating data to a source model), and model generation (generating the underlying cognitive model). We performed these tasks across two cognitive domains - decision making and learning. In the case of data simulated from canonical cognitive models, we found that the LLM successfully identified and generated the ground truth model. In the case of human data, where behavioral noise and lack of knowledge of the true underlying process pose significant challenges, the LLM generated models that are identical or close to the winning model from cognitive science literature. Our findings suggest that LLMs can have a transformative impact on cognitive modeling. With this project, we aim to contribute to an ongoing effort of automating scientific discovery in cognitive science.

Related papers

• The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories [2.6549754445378344]
  We discuss challenges to the use of PLMs as cognitive science theories. We review assumptions used by researchers to map measures of PLM performance to measures of human performance. We end by enumerating criteria for using PLMs as credible accounts of cognition and cognitive development.
  arXiv  Detail & Related papers  (2025-01-22T05:24:23Z)
• Applying Large Language Models in Knowledge Graph-based Enterprise Modeling: Challenges and Opportunities [0.0]
  Large language models (LLMs) in enterprise modeling have recently started to shift from academic research to that of industrial applications. In this paper we employ a knowledge graph-based approach for enterprise modeling and investigate the potential benefits of LLMs.
  arXiv  Detail & Related papers  (2025-01-07T06:34:17Z)
• LLM-based Cognitive Models of Students with Misconceptions [55.29525439159345]
  This paper investigates whether Large Language Models (LLMs) can be instruction-tuned to meet this dual requirement. We introduce MalAlgoPy, a novel Python library that generates datasets reflecting authentic student solution patterns. Our insights enhance our understanding of AI-based student models and pave the way for effective adaptive learning systems.
  arXiv  Detail & Related papers  (2024-10-16T06:51:09Z)
• Cognitive LLMs: Towards Integrating Cognitive Architectures and Large Language Models for Manufacturing Decision-making [51.737762570776006]
  LLM-ACTR is a novel neuro-symbolic architecture that provides human-aligned and versatile decision-making. Our framework extracts and embeds knowledge of ACT-R's internal decision-making process as latent neural representations. Our experiments on novel Design for Manufacturing tasks show both improved task performance as well as improved grounded decision-making capability.
  arXiv  Detail & Related papers  (2024-08-17T11:49:53Z)
• Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice [4.029252551781513]
  We propose a novel way to enhance the utility of Large Language Models as cognitive models. We show that an LLM pretrained on an ecologically valid arithmetic dataset, predicts human behavior better than many traditional cognitive models.
  arXiv  Detail & Related papers  (2024-05-29T17:37:14Z)
• Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension [63.330262740414646]
  We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs) We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
  arXiv  Detail & Related papers  (2024-02-28T04:56:21Z)
• Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach [50.125704610228254]
  Large Language Models (LLMs) have not only exhibited exceptional performance across various tasks, but also demonstrated sparks of intelligence. Recent studies have focused on assessing their capabilities on human exams and revealed their impressive competence in different domains. We conduct an evaluation using MoocRadar, a meticulously annotated human test dataset based on Bloom taxonomy.
  arXiv  Detail & Related papers  (2023-10-12T09:55:45Z)
• Turning large language models into cognitive models [0.0]
  We show that large language models can be turned into cognitive models. These models offer accurate representations of human behavior, even outperforming traditional cognitive models in two decision-making domains. Taken together, these results suggest that large, pre-trained models can be adapted to become generalist cognitive models.
  arXiv  Detail & Related papers  (2023-06-06T18:00:01Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Anti-Retroactive Interference for Lifelong Learning [65.50683752919089]
  We design a paradigm for lifelong learning based on meta-learning and associative mechanism of the brain. It tackles the problem from two aspects: extracting knowledge and memorizing knowledge. It is theoretically analyzed that the proposed learning paradigm can make the models of different tasks converge to the same optimum.
  arXiv  Detail & Related papers  (2022-08-27T09:27:36Z)
• Cognitive Modeling of Semantic Fluency Using Transformers [6.445605125467574]
  We take the first step by predicting human performance in the semantic fluency task (SFT), a well-studied task in cognitive science. We report preliminary evidence suggesting that, despite obvious implementational differences, TLMs can be used to identify individual differences in human fluency task behaviors. We discuss the implications of this work for cognitive modeling of knowledge representations.
  arXiv  Detail & Related papers  (2022-08-20T16:48:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.