GPT-ology, Computational Models, Silicon Sampling: How should we think about LLMs in Cognitive Science?

• URL: http://arxiv.org/abs/2406.09464v1
• Date: Thu, 13 Jun 2024 04:19:17 GMT
• Title: GPT-ology, Computational Models, Silicon Sampling: How should we think about LLMs in Cognitive Science?
• Authors: Desmond C. Ong,
• Abstract summary: We review several emerging research paradigms -- GPT-ology, LLMs-as-computational-models, and silicon sampling" We highlight several outstanding issues about LLMs that have to be addressed to push our science forward.
• Score: 4.242435932138821
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models have taken the cognitive science world by storm. It is perhaps timely now to take stock of the various research paradigms that have been used to make scientific inferences about ``cognition" in these models or about human cognition. We review several emerging research paradigms -- GPT-ology, LLMs-as-computational-models, and ``silicon sampling" -- and review recent papers that have used LLMs under these paradigms. In doing so, we discuss their claims as well as challenges to scientific inference under these various paradigms. We highlight several outstanding issues about LLMs that have to be addressed to push our science forward: closed-source vs open-sourced models; (the lack of visibility of) training data; and reproducibility in LLM research, including forming conventions on new task ``hyperparameters" like instructions and prompts.

Related papers

• 'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants [13.693069737188859]
  Recent excitement around generative models has sparked a wave of proposals suggesting the replacement of human participation and labor in research and development. We conducted interviews with 19 qualitative researchers to understand their perspectives on this paradigm shift.
  arXiv  Detail & Related papers  (2024-09-28T18:28:47Z)
• Towards Efficient Large Language Models for Scientific Text: A Review [4.376712802685017]
  Large language models (LLMs) have ushered in a new era for processing complex information in various fields, including science. Due to the power of LLMs, they require extremely expensive computational resources, intense amounts of data, and training time. In recent years, researchers have proposed various methodologies to make scientific LLMs more affordable.
  arXiv  Detail & Related papers  (2024-08-20T10:57:34Z)
• Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities [89.40778301238642]
  Model merging is an efficient empowerment technique in the machine learning community. There is a significant gap in the literature regarding a systematic and thorough review of these techniques.
  arXiv  Detail & Related papers  (2024-08-14T16:58:48Z)
• Retrieval-Enhanced Machine Learning: Synthesis and Opportunities [60.34182805429511]
  Retrieval-enhancement can be extended to a broader spectrum of machine learning (ML) This work introduces a formal framework of this paradigm, Retrieval-Enhanced Machine Learning (REML), by synthesizing the literature in various domains in ML with consistent notations which is missing from the current literature. The goal of this work is to equip researchers across various disciplines with a comprehensive, formally structured framework of retrieval-enhanced models, thereby fostering interdisciplinary future research.
  arXiv  Detail & Related papers  (2024-07-17T20:01:21Z)
• A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery [68.48094108571432]
  Large language models (LLMs) have revolutionized the way text and other modalities of data are handled. We aim to provide a more holistic view of the research landscape by unveiling cross-field and cross-modal connections between scientific LLMs.
  arXiv  Detail & Related papers  (2024-06-16T08:03:24Z)
• Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension [63.330262740414646]
  We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs) We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
  arXiv  Detail & Related papers  (2024-02-28T04:56:21Z)
• Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs [60.40396361115776]
  This paper introduces a novel collaborative approach, namely SlimPLM, that detects missing knowledge in large language models (LLMs) with a slim proxy model. We employ a proxy model which has far fewer parameters, and take its answers as answers. Heuristic answers are then utilized to predict the knowledge required to answer the user question, as well as the known and unknown knowledge within the LLM.
  arXiv  Detail & Related papers  (2024-02-19T11:11:08Z)
• From Prompt Engineering to Prompt Science With Human in the Loop [12.230632679443364]
  This article presents a new methodology inspired by codebook construction through qualitative methods to address that. We show how a set of researchers can work through a rigorous process of labeling, deliberating, and documenting to remove subjectivity and bring transparency and replicability to prompt generation process.
  arXiv  Detail & Related papers  (2024-01-01T01:37:36Z)
• The Quo Vadis of the Relationship between Language and Large Language Models [3.10770247120758]
  Large Language Models (LLMs) have come to encourage the adoption of LLMs as scientific models of language. We identify the most important theoretical and empirical risks brought about by the adoption of scientific models that lack transparency. We conclude that, at their current stage of development, LLMs hardly offer any explanations for language.
  arXiv  Detail & Related papers  (2023-10-17T10:54:24Z)
• A Comprehensive Overview of Large Language Models [68.22178313875618]
  Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks. This article provides an overview of the existing literature on a broad range of LLM-related concepts.
  arXiv  Detail & Related papers  (2023-07-12T20:01:52Z)
• A Bibliometric Review of Large Language Models Research from 2017 to 2023 [1.4190701053683017]
  Large language models (LLMs) are language models that have demonstrated outstanding performance across a range of natural language processing (NLP) tasks. This paper serves as a roadmap for researchers, practitioners, and policymakers to navigate the current landscape of LLMs research.
  arXiv  Detail & Related papers  (2023-04-03T21:46:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.