Did You Forget What I Asked? Prospective Memory Failures in Large Language Models

• URL: http://arxiv.org/abs/2603.23530v1
• Date: Sat, 07 Mar 2026 05:58:19 GMT
• Title: Did You Forget What I Asked? Prospective Memory Failures in Large Language Models
• Abstract summary: Large language models often fail to satisfy formatting instructions when they must simultaneously perform demanding tasks.<n>We study this behaviour through a prospective memory inspired lens from cognitive psychology, using a controlled paradigm.<n>Vulnerability is highly type-dependent, with drops up to 50%, while avoidance constraints remain comparatively robust.<n>A salience-enhanced format (explicit instruction framing plus a trailing reminder) recovers much of the lost compliance, restoring performance to 90-100% in many settings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models often fail to satisfy formatting instructions when they must simultaneously perform demanding tasks. We study this behaviour through a prospective memory inspired lens from cognitive psychology, using a controlled paradigm that combines verifiable formatting constraints with benchmark tasks of increasing complexity. Across three model families and over 8,000 prompts, compliance drops by 2-21% under concurrent task load. Vulnerability is highly type-dependent: terminal constraints (requiring action at the response boundary) degrade most, with drops up to 50%, while avoidance constraints remain comparatively robust. A salience-enhanced format (explicit instruction framing plus a trailing reminder) recovers much of the lost compliance, restoring performance to 90-100% in many settings. Interference is bidirectional: formatting constraints can also reduce task accuracy, with one model's GSM8K accuracy dropping from 93% to 27%. In additional stacking experiments, joint compliance declines sharply as constraints accumulate. All results use deterministic programmatic checkers without an LLM-as-judge component on publicly available datasets.

Related papers

• Constraint-Guided Enterprise Data Mapping with Large Language Models [0.25453602861405583]
  We propose a neuro-symbolic method with three stages: (i) schema-grounded admissibility constraints with metadata mc = tau_c, delta_c>, (ii) constraint-restricted candidate generation with cascade relaxation to guarantee a nonempty feasible set under noise, and (iii) neural ranking with bounded disambiguation restricted to that feasible set.<n> Methodologically, constraints operate as hypothesis-space operators rather than post-hoc validators, enabling controlled degradation under relaxation and auditable, human-guidable decisions.
  arXiv  Detail & Related papers  (2026-08-25T08:23:45Z)
• Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction [2.668746022268522]
  Large language models are increasingly deployed in settings that require simultaneous adherence to multiple explicit constraints.<n>Individual constraints are handled proficiently, but the compositional regime remains poorly characterized.<n>We introduce Constraint Saturation Evaluation (CSE), a benchmark that systematically varies the number of simultaneous constraints.
  arXiv  Detail & Related papers  (2026-08-12T10:57:06Z)
• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation [0.0]
  We introduce DriftBench, a benchmark for evaluating constraint adherence in scientific ideation.<n>We find that iterative pressure reliably increases structural complexity and often reduces adherence to original constraints.<n>We release all briefs, prompts, rubrics, transcripts, and scores as an open benchmark.
  arXiv  Detail & Related papers  (2026-04-30T15:46:33Z)
• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• Compact Constraint Encoding for LLM Code Generation: An Empirical Study of Token Economics and Constraint Compliance [0.0]
  We investigate whether compact, structured constraint headers can reduce prompt token consumption without degrading constraint compliance.<n>We find that compact headers reduce constraint-portion tokens by approximately 71% and full-prompt tokens by 25-30%, replicated across three independent rounds.
  arXiv  Detail & Related papers  (2026-04-08T15:18:33Z)
• CCTU: A Benchmark for Tool Use under Complex Constraints [66.87622847854337]
  We introduce CCTU, a benchmark for evaluating large language models (LLMs) under complex constraints.<n>The benchmark comprises 200 carefully curated and challenging test cases across diverse tool-use scenarios.<n>We develop an executable constraint validation module that performs step-level validation and enforces compliance.
  arXiv  Detail & Related papers  (2026-03-16T14:05:13Z)
• Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations [0.0]
  We study failure to execute a verifiable, prompt-grounded specification even when the correct value is present in context.<n>Specifically, failures decompose into Stage2A (gating) errors, where the model does not enter answer mode, and Stage2B (binding) errors, where it enters answer mode but selects the wrong candidate.
  arXiv  Detail & Related papers  (2026-02-22T15:43:41Z)
• GLEAN: Grounded Lightweight Evaluation Anchors for Contamination-Aware Tabular Reasoning [0.5414847001704249]
  We propose a lightweight evaluation protocol that integrates contamination-aware probes, weak-supervision governance, retrieval-reasoning diagnostics, and structured error attribution.<n>We evaluate across TabFact, WTQ via Squall, TableBench, RobuT, and SciTab under a 16GB GPU budget.
  arXiv  Detail & Related papers  (2026-01-22T13:56:19Z)
• MERGETUNE: Continued fine-tuning of vision-language models [77.8627788911249]
  Fine-tuning vision-language models (VLMs) often leads to catastrophic forgetting of pretrained knowledge.<n>We introduce a novel paradigm, continued fine-tuning (CFT), which seeks to recover pretrained knowledge after a zero-shot model has already been adapted.
  arXiv  Detail & Related papers  (2026-01-15T15:15:53Z)
• Programming over Thinking: Efficient and Robust Multi-Constraint Planning [54.77940831026738]
  SCOPE is a framework that disentangles query-specific reasoning from generic code execution.<n>SCOPE achieves state-of-the-art performance while lowering cost and latency.
  arXiv  Detail & Related papers  (2026-01-14T02:58:07Z)
• CIFE: Code Instruction-Following Evaluation [3.941243815951084]
  We introduce a benchmark of 1,000 Python tasks, each paired with an average of 7 developer-specified constraints spanning 13 categories.<n>We evaluate 14 open- and closed-source models using complementary adherence metrics and propose the C2A Score, a composite measure that jointly captures correctness and constraint compliance.<n>Results reveal a substantial gap between partial and strict satisfaction, while strong models achieve over 90% partial adherence, strict adherence remains between 39-66%.
  arXiv  Detail & Related papers  (2025-12-19T09:43:20Z)
• Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models [2.3991974633684854]
  Large language models must satisfy hard orthographic constraints during controlled text generation.<n>We evaluate 28 configurations spanning three model families on 58 word puzzles requiring character-level constraint satisfaction.
  arXiv  Detail & Related papers  (2025-11-26T06:12:33Z)
• CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs [62.116710797795314]
  Large Language Models (LLMs) increasingly use persistent memory from past interactions to enhance personalization and task performance.<n>We present CIMemories, a benchmark for evaluating whether LLMs appropriately control information flow from memory based on task context.
  arXiv  Detail & Related papers  (2025-11-18T21:51:23Z)
• A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models [48.361839372110246]
  We develop an automated instruction generation pipeline that performs constraint expansion, conflict detection, and instruction rewriting.<n>We evaluate 19 large language models and uncover substantial variation in performance across constraint forms.<n>In-depth analysis indicates that these gains stem primarily from modifications in the model's attention modules parameters.
  arXiv  Detail & Related papers  (2025-05-12T14:16:55Z)
• Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models [38.79074982172423]
  We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text. We propose modeling factual queries as constraint satisfaction problems. We find a strong positive relationship between the LLM's attention to constraint tokens and the factual accuracy of generations.
  arXiv  Detail & Related papers  (2023-09-26T17:48:55Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.