SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory

• URL: http://arxiv.org/abs/2605.15710v1
• Date: Fri, 15 May 2026 08:00:46 GMT
• Title: SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory
• Abstract summary: Existing benchmarks for multimodal memory reasoning largely evaluate systems within pre-assembled contexts.<n>We argue that source-distributed memory composition is an important and under-examined bottleneck in multimodal agent memory.<n>We introduce Source-distributed Multimodal Memory Benchmark which measures whether agents can retrieve, align, and compose multimodal evidence scattered across multiple sources.
• Score: 34.330518284737806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing benchmarks for multimodal memory reasoning largely evaluate systems within pre-assembled contexts, but under-evaluate whether agents can use evidence distributed across independently originated sources. We argue that source-distributed memory composition is an important and under-examined bottleneck in multimodal agent memory, especially when relevant evidence is fragmented across heterogeneous artifacts such as conversations, profiles, screenshots, tables, images, and documents. To address this gap, we introduce Source-distributed Multimodal Memory Benchmark(SMMBench), which measures whether agents can retrieve, align, and compose multimodal evidence scattered across multiple sources rather than reason within a single curated context. SMMBench evaluates four core capabilities: (1) cross-source multimodal reasoning; (2) conflict resolution; (3) preference reasoning; (4) memory-grounded action prediction. The benchmark contains 1877 samples grounded in 264 sources. Experiments on representative memory-style and retrieval-based baselines show that current systems still struggle on these capabilities, positioning source-distributed multimodal memory as an important and still under-evaluated challenge for multimodal agents. Our data are available at https://huggingface.co/datasets/HuacanChai/SMMBench.

Related papers

• GraphMemix: Query-Aware Evidence Forests for Long-Term Multimodal Agent Memory [87.58855368314693]
  We propose GraphMemix, a graph memory framework that models memory organization as query-aware evidence-forest construction.<n> Experimental results across four long-term multimodal memory benchmarks demonstrate significant improvements with different foundation models.
  arXiv  Detail & Related papers  (2026-08-27T11:28:49Z)
• Beyond Memory Majority: Latent-Source Reasoning for Multi-Agent Memory Arbitration [1.1803729553124735]
  Existing memory methods treat retrieved memories as independent evidence and combine them through voting or weighting.<n> memories written by different agents may inherit the same upstream source or shared bias, causing correlated evidence to be repeatedly counted and creating a false majority.<n>We propose textbfCorrelation-textbfAware textbfMemory textbfArbitration (CAMA) framework that jointly decouples retrieved memories and recovers missing independent evidence.
  arXiv  Detail & Related papers  (2026-08-20T06:50:23Z)
• MemFuse: Multi-Source Memory Fusion from Fragmented Observations [32.62990579442196]
  We introduce **MemFuseBench**, a benchmark for *multi-source memory fusion*.<n>MemFuseBench is built with a Scene-to-Sensor pipeline that synthesizes controllable scenarios into source-tagged observations, evidence-grounded questions, and adversarial distractors.<n>We propose **MemFuse**, a structured memory system that preserves source-level evidence in event-layer atomic memory and organizes related atomic events into cluster-layer fused memory within a causal graph.
  arXiv  Detail & Related papers  (2026-08-19T09:04:07Z)
• M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions [21.009839718901855]
  M$3$Exam is a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction.<n>We propose M$3$Proctor, a multimodal memory method that detects query modality bias and consumes raw visual sources only on demand.
  arXiv  Detail & Related papers  (2026-06-05T15:44:18Z)
• M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks [19.25978075323521]
  M$3$Eval is the first comprehensive evaluation framework and benchmark for probing different memory dimensions in multi-modal models.<n>We conduct experiments across representative multi-modal models, revealing consistent weaknesses and distinctive behaviors.<n>Our findings highlight memory as a fundamental yet underexplored capability and offer insights for designing more effective memory mechanisms in multi-modal models.
  arXiv  Detail & Related papers  (2026-06-03T15:28:57Z)
• MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models [50.25006399944962]
  Memory is essential for large vision-language models to handle long, multimodal interactions.<n>MEMLENS is a benchmark for memory in multimodal multi-session conversations.<n>We evaluate 27 LVLMs and 7 memory-augmented agents.
  arXiv  Detail & Related papers  (2026-05-14T14:41:17Z)
• GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations [25.703133924514884]
  Large Language Model (LLM) agents increasingly serve as personal assistants and workplace collaborators.<n>Existing memory systems and benchmarks are built around the dyadic, single-user setup.<n>We introduce GroupMemBench, a benchmark that exposes three properties of group memory.
  arXiv  Detail & Related papers  (2026-05-14T07:38:29Z)
• MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents [57.32877731797049]
  MultiHaystack is the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions.<n>We find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus.
  arXiv  Detail & Related papers  (2026-03-05T21:43:02Z)
• According to Me: Long-Term Personalized Referential Memory QA [27.402232752643275]
  ATM-Bench is the first benchmark for multimodal, multi-source personalized referential Memory QA.<n>Guided Memory (SGM) structurally represents memory items originated from different sources.<n>We find poor performance (under 20% accuracy) on the ATM-Bench-Hard set.
  arXiv  Detail & Related papers  (2026-03-02T15:42:29Z)
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG [82.84014669683863]
  Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models to real-world knowledge bases.<n>UniDoc-Bench is the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages.<n>Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval.
  arXiv  Detail & Related papers  (2025-10-04T04:30:13Z)
• SEDM: Scalable Self-Evolving Distributed Memory for Agents [23.182291416527764]
  SEDM is a verifiable and adaptive framework that transforms memory from a passive repository into an active, self-optimizing component.<n>We show that SEDM improves reasoning accuracy while reducing token overhead compared with strong memory baselines.<n>Results highlight SEDM as a scalable and sustainable memory mechanism for open-ended multi-agent collaboration.
  arXiv  Detail & Related papers  (2025-09-11T14:37:37Z)
• Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering [60.062194349648195]
  Document Visual Question Answering (DocVQA) faces dual challenges in processing lengthy multimodal documents.<n>Current document retrieval-augmented generation (DocRAG) methods remain limited by their text-centric approaches.<n>We introduce MMDocRAG, a comprehensive benchmark featuring 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains.
  arXiv  Detail & Related papers  (2025-05-22T09:52:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.