Beyond the Shadows of Plato's Cave: Evaluating False Memory in Autonomous Agents via Counterfactual Reasoning
Abstract Overview
This paper studies false memory in autonomous agents, defined as cases where memories formed from prior experience lead to inappropriate beliefs in new situations. The authors introduce FAME, a training-free evaluation framework that uses counterfactual queries to measure how an agent's latent memory-induced concept shifts in hidden-state space. The method is designed to distinguish faithful memory from false memory without relying on reward design, fine-tuning, or answer generation. The paper also presents a taxonomy of false memory covering spurious correlation, environment shift, and knowledge conflict, together with theoretical foundations for counterfactual concept-drift evaluation.
Novelty
The main novelty is framing false memory as an internal-belief evaluation problem and assessing it through the counterfactual evolution of memory-induced concepts rather than surface outputs. FAME is distinctive in using hidden-state concept drift as an answer-free, training-free signal for false memory across multiple scenario types.
Results
In controlled false-memory settings, FAME achieves AUROCs of 76.2% to 96.7% and consistently outperforms all evaluated baselines. On realistic benchmarks spanning GSM-Symbolic, GitChameleon, and BigBench-Hard, it exceeds the best baseline by 3.4% to 23.3% AUROC. The experiments also reveal that observing answer confidence alone is often insufficient to detect false memory, whereas hidden-state concept drift strongly correlates with behavioral margin shifts.
Key Points
- FAME evaluates false memory by comparing factual and counterfactual hidden-state concept representations induced by memory and queries before answer generation.
- The framework categorizes and evaluates three practical sources of false memory: spurious correlation, environment shift, and knowledge conflict.
- Empirical analyses show that boundary-crossing concept drift is more informative than drift magnitude alone, and that false memory often cannot be distinguished solely from output answers.