IFCMemoryBench: Evaluating Long-Term Memory of LLM-Based Agents in BIM Information Retrieval

• URL: http://arxiv.org/abs/2607.26072v1
• Date: Mon, 13 Jul 2026 08:45:00 GMT
• Title: IFCMemoryBench: Evaluating Long-Term Memory of LLM-Based Agents in BIM Information Retrieval
• Abstract summary: We introduce IFCMemoryBench, a benchmark for evaluating long-term memory in LLM-based information retrieval.<n> IFCMemoryBench contains 143 multi-session tasks across 19 projects and 4,016 prior sessions, derived from incomplete-information questions in IFC-Bench v2.<n>Our evaluation framework decomposes memory performance into ingestion, retrieval, and utilization, and measures both answer quality and memory quality with expert-validated LLM judges.
• Score: 39.66871834776678
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Long-term memory is becoming a core capability of LLM-based agents, but existing evaluations largely test conversational recall in open-domain or persona-grounded settings. We argue that a stronger test is whether an agent can reuse information from prior sessions while acting over a live, structured, domain-specific environment. We study this problem in Building Information Modelling (BIM), a professional engineering workflow where agents must query large IFC models while also relying on project specifications, client decisions, and engineering conventions often discussed in conversation but absent from the model. We introduce IFCMemoryBench, a benchmark for evaluating long-term memory in LLM-based BIM information retrieval. IFCMemoryBench contains 143 multi-session tasks across 19 projects and 4,016 prior sessions, derived from incomplete-information questions in IFC-Bench v2. Each task seeds missing project context across earlier conversations and later asks a probe question that can be answered only by combining remembered context with live IFC queries. Our evaluation framework decomposes memory performance into ingestion, retrieval, and utilization, and measures both answer quality and memory quality with expert-validated LLM judges. We evaluate representative vector-, graph-, and file-based memory systems. The strongest system achieves only 32.4% answer accuracy under a deployment-realistic ingestion scope, and remains below 60% under oracle-filtered ingestion or a stronger probe agent. Analysis shows that current general-purpose memory systems often retrieve topically relevant context but store project knowledge as incomplete or fragmented facts. These results reveal a domain-transfer gap in agent memory and suggest that reliable professional agents require domain-aware memory representations linking conversations, project knowledge, and structured model entities.

Related papers

• MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery [58.64329475683699]
  We argue that long-term memory should be evaluated as an auditable post-interaction artifact.<n>We instantiate this view in MEMPROBE, a benchmark in which a memory-equipped agent assists simulated users.<n>We find that successful assistance and recoverable memory behave as distinct capabilities.
  arXiv  Detail & Related papers  (2026-06-23T13:52:46Z)
• TA-Mem: Tool-Augmented Autonomous Memory Retrieval for LLM in Long-Term Conversational QA [8.036549927091286]
  Large Language Model (LLM) has exhibited strong reasoning ability in text-based contexts across various domains.<n>The limitation of context window poses challenges for the model on long-range inference tasks.<n>We introduce a tool-augmented autonomous memory retrieval framework (TA-Mem)<n>The TA-Mem is evaluated on the LoCoMo dataset, achieving significant performance improvements over existing baseline approaches.
  arXiv  Detail & Related papers  (2026-03-10T07:27:01Z)
• MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks [55.145729491377374]
  Existing evaluations of agents with memory typically assess memorization and action in isolation.<n>We introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops.<n> MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning.
  arXiv  Detail & Related papers  (2026-02-18T09:49:14Z)
• PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents [74.4491017652226]
  We propose PlugMem, a task-agnostic memory module that can be attached to arbitrary LLM agents without task-specific redesign.<n>Motivated by the fact that decision-relevant information is concentrated as abstract knowledge rather than raw experience, we draw on cognitive science to structure episodic memories into a compact, knowledge-centric memory graph.<n>We evaluate PlugMem unchanged across three benchmarks (long-horizon conversational question answering, multi-hop knowledge retrieval, and web agent tasks)
  arXiv  Detail & Related papers  (2026-02-06T18:55:38Z)
• Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents [76.76004970226485]
  Long-term memory is a critical capability for multimodal large language model (MLLM) agents.<n>Mem-Gallery is a new benchmark for evaluating multimodal long-term conversational memory in MLLM agents.
  arXiv  Detail & Related papers  (2026-01-07T02:03:13Z)
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory [89.65731902036669]
  Evo-Memory is a streaming benchmark and framework for evaluating self-evolving memory in large language model (LLM) agents.<n>We evaluate over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets.
  arXiv  Detail & Related papers  (2025-11-25T21:08:07Z)
• MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments [6.12783571098263]
  MEMTRACK is a benchmark designed to evaluate long-term memory and state tracking in multi-platform agent environments.<n>Each benchmark instance provides a chronologically platform-interleaved timeline, with noisy, conflicting, cross-referring information.<n>Our benchmark tests memory capabilities such as acquistion, selection and conflict resolution.
  arXiv  Detail & Related papers  (2025-10-01T18:34:03Z)
• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions [22.190297901876278]
  We identify four core competencies essential for memory agents: accurate retrieval, test-time learning, long-range understanding, and selective forgetting.<n>Existing benchmarks either rely on limited context lengths or are tailored for static, long-context settings like book-based QA.<n>We introduce MemoryAgentBench, a new benchmark specifically designed for memory agents.
  arXiv  Detail & Related papers  (2025-07-07T17:59:54Z)
• MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents [26.647812147336538]
  We construct a more comprehensive dataset and benchmark to evaluate the memory capability of LLM-based agents.<n>Our dataset incorporates factual memory and reflective memory as different levels, and proposes participation and observation as various interactive scenarios.<n>Based on our dataset, we present a benchmark, named MemBench, to evaluate the memory capability of LLM-based agents from multiple aspects, including their effectiveness, efficiency, and capacity.
  arXiv  Detail & Related papers  (2025-06-20T10:09:23Z)
• Memory Sharing for Large Language Model based Agents [43.53494041932615]
  This paper introduces the Memory Sharing, a framework which integrates the real-time memory filter, storage and retrieval to enhance the In-Context Learning process. The experimental results demonstrate that the MS framework significantly improves the agents' performance in addressing open-ended questions.
  arXiv  Detail & Related papers  (2024-04-15T17:57:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.