Language Models Do Not Have Human-Like Working Memory

• URL: http://arxiv.org/abs/2505.10571v3
• Date: Tue, 23 Sep 2025 17:47:04 GMT
• Title: Language Models Do Not Have Human-Like Working Memory
• Abstract summary: We show that Large Language Models (LLMs) lack a fundamental aspect of human cognition: working memory.<n>Human working memory is an active cognitive system that enables coherent reasoning and decision-making.<n>Our work establishes a new benchmark for evaluating working memory in LLMs.
• Score: 24.355006642653503
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Large Language Models (LLMs) exhibit remarkable reasoning abilities, we demonstrate that they lack a fundamental aspect of human cognition: working memory. Human working memory is an active cognitive system that enables not only the temporary storage of information but also its processing and utilization, enabling coherent reasoning and decision-making. Without working memory, individuals may produce unrealistic responses, exhibit self-contradictions, and struggle with tasks that require mental reasoning. Existing evaluations using N-back or context-dependent tasks fall short as they allow LLMs to exploit external context rather than retaining the reasoning process in the latent space. We introduce three novel tasks: (1) Number Guessing, (2) Yes-No Deduction, and (3) Math Magic, designed to isolate internal representation from external context. Across seventeen frontier models spanning four major model families, we consistently observe irrational or contradictory behaviors, indicating LLMs' inability to retain and manipulate latent information. Our work establishes a new benchmark for evaluating working memory in LLMs and highlights this limitation as a key bottleneck for advancing reliable reasoning systems. Code and prompts for the experiments are available at https://github.com/penguinnnnn/LLM-Working-Memory.

Related papers

• Cognitive Workspace: Active Memory Management for LLMs -- An Empirical Study of Functional Infinite Context [0.0]
  Large Language Models (LLMs) face fundamental limitations in context management.<n>We propose Cognitive Workspace, a novel paradigm that transcends traditional Retrieval-Augmented Generation (RAG)<n> Cognitive Workspace addresses these limitations through three core innovations: (1) active memory management with deliberate information curation, (2) hierarchical cognitive buffers enabling persistent working states, and (3) task-driven context optimization.
  arXiv  Detail & Related papers  (2025-08-08T16:32:47Z)
• The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Direction [34.86855316803838]
  We identify a set of linear features in the model's residual stream that govern the balance between genuine reasoning and memory recall.<n>We show that intervening in these reasoning features helps the model more accurately activate the most relevant problem-solving capabilities during answer generation.
  arXiv  Detail & Related papers  (2025-03-29T14:00:44Z)
• General Intelligence Requires Reward-based Pretraining [14.057301560895505]
  Large Language Models (LLMs) have demonstrated impressive real-world utility.<n>But their ability to reason adaptively and robustly remains fragile.<n>We propose disangling knowledge and reasoning through three key directions.
  arXiv  Detail & Related papers  (2025-02-26T18:51:12Z)
• Do Language Models Understand the Cognitive Tasks Given to Them? Investigations with the N-Back Paradigm [9.577716124021029]
  We show that GPT 3.5's declining performance on 2-back and 3-back tasks reflects a working memory capacity limit similar to humans.<n>By analyzing a range of open-source language models of varying performance levels on these tasks, we show that the poor performance is due at least in part to a limitation in task comprehension and task set maintenance.
  arXiv  Detail & Related papers  (2024-12-24T03:06:52Z)
• Judgment of Learning: A Human Ability Beyond Generative Artificial Intelligence [0.0]
  Large language models (LLMs) increasingly mimic human cognition in various language-based tasks.<n>We introduce a cross-agent prediction model to assess whether ChatGPT-based LLMs align with human judgments of learning (JOL)<n>Our results revealed that while human JOL reliably predicted actual memory performance, none of the tested LLMs demonstrated comparable predictive accuracy.
  arXiv  Detail & Related papers  (2024-10-17T09:42:30Z)
• UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models [12.45822383965784]
  We introduce UnDIAL (Unlearning via Self-Distillation on Adjusted Logits), a novel and robust unlearning method. Our approach leverages self-distillation to adjust logits and selectively reduce the influence of targeted tokens.
  arXiv  Detail & Related papers  (2024-02-15T16:21:14Z)
• CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
  We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
  arXiv  Detail & Related papers  (2023-11-29T08:29:54Z)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [82.7042006247124]
  We show that even the most capable AI models reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
  arXiv  Detail & Related papers  (2023-10-27T04:15:30Z)
• Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Models [83.63242931107638]
  We propose four characteristics of generally intelligent agents. We argue that active engagement with objects in the real world delivers more robust signals for forming conceptual representations. We conclude by outlining promising future research directions in the field of artificial general intelligence.
  arXiv  Detail & Related papers  (2023-07-07T13:58:16Z)
• Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners [75.85554779782048]
  Large Language Models (LLMs) have excited the natural language and machine learning community over recent years. Despite of numerous successful applications, the underlying mechanism of such in-context capabilities still remains unclear. In this work, we hypothesize that the learned textitsemantics of language tokens do the most heavy lifting during the reasoning process.
  arXiv  Detail & Related papers  (2023-05-24T07:33:34Z)
• RET-LLM: Towards a General Read-Write Memory for Large Language Models [53.288356721954514]
  RET-LLM is a novel framework that equips large language models with a general write-read memory unit. Inspired by Davidsonian semantics theory, we extract and save knowledge in the form of triplets. Our framework exhibits robust performance in handling temporal-based question answering tasks.
  arXiv  Detail & Related papers  (2023-05-23T17:53:38Z)
• ReAct: Synergizing Reasoning and Acting in Language Models [44.746116256516046]
  We show that large language models (LLMs) can generate both reasoning traces and task-specific actions in an interleaved manner. We apply our approach, named ReAct, to a diverse set of language and decision making tasks. ReAct overcomes issues of hallucination and error propagation prevalent in chain-of-thought reasoning by interacting with a simple Wikipedia API.
  arXiv  Detail & Related papers  (2022-10-06T01:00:32Z)
• oLMpics -- On what Language Model Pre-training Captures [84.60594612120173]
  We propose eight reasoning tasks, which require operations such as comparison, conjunction, and composition. A fundamental challenge is to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data.
  arXiv  Detail & Related papers  (2019-12-31T12:11:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.