Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agents

• URL: http://arxiv.org/abs/2603.23013v1
• Date: Tue, 24 Mar 2026 09:55:11 GMT
• Title: Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agents
• Abstract summary: Production AI agents frequently receive user-specific queries that are highly repetitive, with up to 47% semantically similar to prior interactions.<n>We argue that this redundancy can be exploited through conversational memory, transforming repetition from a cost burden into an efficiency advantage.
• Score: 9.457255218406333
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Production AI agents frequently receive user-specific queries that are highly repetitive, with up to 47\% being semantically similar to prior interactions, yet each query is typically processed with the same computational cost. We argue that this redundancy can be exploited through conversational memory, transforming repetition from a cost burden into an efficiency advantage. We propose a memory-augmented inference framework in which a lightweight 8B-parameter model leverages retrieved conversational context to answer all queries via a low-cost inference path. Without any additional training or labeled data, this approach achieves 30.5\% F1, recovering 69\% of the performance of a full-context 235B model while reducing effective cost by 96\%. Notably, a 235B model without memory (13.7\% F1) underperforms even the standalone 8B model (15.4\% F1), indicating that for user-specific queries, access to relevant knowledge outweighs model scale. We further analyze the role of routing and confidence. At practical confidence thresholds, routing alone already directs 96\% of queries to the small model, but yields poor accuracy (13.0\% F1) due to confident hallucinations. Memory does not substantially alter routing decisions; instead, it improves correctness by grounding responses in retrieved user-specific information. As conversational memory accumulates over time, coverage of recurring topics increases, further narrowing the performance gap. We evaluate on 152 LoCoMo questions (Qwen3-8B/235B) and 500 LongMemEval questions. Incorporating hybrid retrieval (BM25 + cosine similarity) improves performance by an additional +7.7 F1, demonstrating that retrieval quality directly enhances end-to-end system performance. Overall, our results highlight that memory, rather than model size, is the primary driver of accuracy and efficiency in persistent AI agents.

Related papers

• MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval [37.54115468116941]
  This report introduces MemReranker, a reranking model family (0.6B/4B) built on Qwen3-Reranker through multi-stage LLM knowledge distillation.<n>On the memory retrieval benchmark, MemReranker-0.6B substantially outperforms BGE-Reranker and matches open-source 4B/8B models as well as GPT-4o-mini on key metrics.<n>MemReranker-4B further achieves 0.737 MAP, with several metrics on par with Gemini-3-Flash, while maintaining inference latency at only 10-20% of large models.
  arXiv  Detail & Related papers  (2026-05-07T12:33:57Z)
• From Unstructured Recall to Schema-Grounded Memory: Reliable AI Memory via Iterative, Schema-Aware Extraction [39.146761527401424]
  Persistent AI memory is often reduced to a retrieval problem.<n>This paper argues that reliable external AI memory must be schema-grounded harnesses.<n>We present an iterative, schema-aware write path that decomposes memory ingestion into object detection, field detection, and field-value extraction.
  arXiv  Detail & Related papers  (2026-04-30T14:14:02Z)
• MemMachine: A Ground-Truth-Preserving Memory System for Personalized AI Agents [2.541923091180284]
  Large Language Model (LLM) agents require persistent memory to maintain personalization, factual continuity, and long-horizon reasoning.<n>We present MemMachine, an open-source memory system that integrates short-term, long-term episodic, and profile memory.<n>MemMachine uses contextualized retrieval that expands nucleus matches with surrounding context, improving recall when relevant evidence spans multiple dialogue turns.
  arXiv  Detail & Related papers  (2026-04-06T16:57:06Z)
• CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs [62.116710797795314]
  Large Language Models (LLMs) increasingly use persistent memory from past interactions to enhance personalization and task performance.<n>We present CIMemories, a benchmark for evaluating whether LLMs appropriately control information flow from memory based on task context.
  arXiv  Detail & Related papers  (2025-11-18T21:51:23Z)
• MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning [73.27233666920618]
  We propose MemSearcher, an agent workflow that iteratively maintains a compact memory and combines the current turn with it.<n>At each turn, MemSearcher fuses the user's question with the memory to generate reasoning traces, perform search actions, and update memory to retain only information essential for solving the task.<n>We introduce multi-context GRPO, an end-to-end RL framework that jointly optimize reasoning, search strategies, and memory management of MemSearcher Agents.
  arXiv  Detail & Related papers  (2025-11-04T18:27:39Z)
• Continual Learning via Sparse Memory Finetuning [58.163704181154834]
  We investigate whether sparse parameter updates can enable learning without catastrophic forgetting.<n>By updating only the memory slots that are highly activated by a new piece of knowledge relative to usage on pretraining data, we reduce interference between new knowledge and the model's existing capabilities.<n>We find that sparse memory finetuning learns new knowledge while exhibiting substantially less forgetting.
  arXiv  Detail & Related papers  (2025-10-16T19:44:38Z)
• MLP Memory: A Retriever-Pretrained Memory for Large Language Models [39.890098399493986]
  NLPRAG Memory is a lightweight parametric module that learns to internalize retrieval patterns without explicit document access.<n>Our architecture integrates this pretrained Memory with Transformer decoders through simple probability, yielding 17.5% and 24.1% scaling gains on WikiText-103 and Web datasets, respectively.
  arXiv  Detail & Related papers  (2025-08-03T16:40:53Z)
• Maximally-Informative Retrieval for State Space Model Generation [59.954191072042526]
  We introduce Retrieval In-Context Optimization (RICO) to minimize model uncertainty for a particular query at test-time.<n>Unlike traditional retrieval-augmented generation (RAG), which relies on externals for document retrieval, our approach leverages direct feedback from the model.<n>We show that standard top-$k$ retrieval with model gradients can approximate our optimization procedure, and provide connections to the leave-one-out loss.
  arXiv  Detail & Related papers  (2025-06-13T18:08:54Z)
• Efficient and Accurate Memorable Conversation Model using DPO based on sLLM [0.1285666675142425]
  This paper presents a conversation model that efficiently manages memory as sessions progress. Our model using DPO algorithm shows an improvement about 0.0591 of BERTScore in memory accuracy. Our model demonstrates efficiency not only in terms of accuracy but also in resource utilization.
  arXiv  Detail & Related papers  (2024-07-09T04:17:39Z)
• Efficient Prompt Caching via Embedding Similarity [26.456212783693545]
  We focus on the prediction accuracy of prompt caching for single-round question-answering tasks via embedding similarity. We propose a distillation-based method to fine-tune the existing embeddings for better better prediction. We also conduct simulations demonstrating that our trained models achieve better caching efficiency than the previous embedding model.
  arXiv  Detail & Related papers  (2024-02-02T06:34:11Z)
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems [63.713297451300086]
  We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B. Their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
  arXiv  Detail & Related papers  (2022-06-15T20:44:23Z)
• AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction [75.16836697734995]
  We propose a two-stage algorithm called Automatic Feature Interaction Selection (AutoFIS) AutoFIS can automatically identify important feature interactions for factorization models with computational cost just equivalent to training the target model to convergence. AutoFIS has been deployed onto the training platform of Huawei App Store recommendation service.
  arXiv  Detail & Related papers  (2020-03-25T06:53:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.