H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions

• URL: http://arxiv.org/abs/2606.09461v1
• Date: Mon, 08 Jun 2026 13:17:16 GMT
• Title: H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions
• Abstract summary: We introduce H2HMem, a Human-to-Human Multimodal Memory Benchmark for evaluating memory capabilities in complex human-human interactions.<n>H2HMem includes both dyadic and multi-party conversations with multimodal information streams, and evaluates agents along three dimensions: memory recall, reasoning, and application.
• Score: 60.50162125179037
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model agents are increasingly deployed in human-human interaction settings, such as meeting assistants and clinical documentation systems, where they must observe conversations and retain information for downstream queries. Unlike traditional human-assistant settings, these environments are inherently multimodal, involve complex discourse phenomena such as anaphora and deixis, and contain asynchronous or conflicting information from multiple participants. However, existing memory benchmarks largely focus on single-user, text-only interactions, failing to capture these challenges. To address this gap, we introduce H2HMem, a Human-to-Human Multimodal Memory Benchmark for evaluating memory capabilities in complex human-human interactions. H2HMem includes both dyadic and multi-party conversations with multimodal information streams, and evaluates agents along three dimensions: memory recall, reasoning, and application. Experiments with advanced agents reveal substantial limitations in constructing, retaining, and utilizing memories across modalities, participants, and sessions, highlighting substantial room for improvement in next-generation LLM agents.

Related papers

• Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory [54.947805187562274]
  We introduce realistic dialogues across diverse interaction scenarios that exhibit dynamic temporal evolution and long-term coherence.<n>The resulting benchmark encompasses challenging question-answer pairs spanning seven inquiry types.<n>We identify at least one of 27 critical memory characteristics that we identify as essential yet underexplored in current research.
  arXiv  Detail & Related papers  (2026-05-29T09:51:22Z)
• Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions [17.9008221917999]
  POLAR is a memory-augmented framework for personalized embodied agents over long-term user interactions.<n>To execute embodied tasks, POLAR retrieves relevant memories to interpret the current request and guide task execution.<n>Results show that the proposed memory mechanism consistently improves performance by enabling more effective use of information accumulated over prior interactions.
  arXiv  Detail & Related papers  (2026-05-25T18:27:27Z)
• M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactions [29.497569352308037]
  M2A is an agentic dual-layer hybrid memory system that maintains personalized multimodal information through online updates.<n>ChatAgent manages user interactions and autonomously decides when to query or update memory.<n>MemoryManager breaks down memory requests from ChatAgent into detailed operations on the dual-layer memory bank.
  arXiv  Detail & Related papers  (2026-02-07T17:13:56Z)
• The AI Hippocampus: How Far are We From Human Memory? [77.04745635827278]
  Implicit memory refers to the knowledge embedded within the internal parameters of pre-trained transformers.<n>Explicit memory involves external storage and retrieval components designed to augment model outputs with dynamic, queryable knowledge representations.<n>Agentic memory introduces persistent, temporally extended memory structures within autonomous agents.
  arXiv  Detail & Related papers  (2026-01-14T03:24:08Z)
• EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory [63.84216832544323]
  EvolMem is a new benchmark for assessing multi-session memory capabilities of large language models (LLMs) and agent systems.<n>To construct the benchmark, we introduce a hybrid data synthesis framework that consists of topic-initiated generation and narrative-inspired transformations.<n>Extensive evaluation reveals that no LLM consistently outperforms others across all memory dimensions.
  arXiv  Detail & Related papers  (2026-01-07T03:14:42Z)
• Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents [76.76004970226485]
  Long-term memory is a critical capability for multimodal large language model (MLLM) agents.<n>Mem-Gallery is a new benchmark for evaluating multimodal long-term conversational memory in MLLM agents.
  arXiv  Detail & Related papers  (2026-01-07T02:03:13Z)
• HiMeS: Hippocampus-inspired Memory System for Personalized AI Assistants [7.477189210398971]
  We propose HiMeS, an AI-assistant architecture that fuses short-term and long-term memory.<n>Inspired by the hippocampus-neocortex memory mechanism, we propose HiMeS, an AI-assistant architecture that fuses short-term and long-term memory.
  arXiv  Detail & Related papers  (2026-01-06T05:05:50Z)
• Human vs. Agent in Task-Oriented Conversations [22.743152820695588]
  This work presents the first systematic comparison between large language models (LLMs)-simulated users and human users in personalized task-oriented conversations.<n>Our analysis reveals significant behavioral differences between the two user types in problem-solving approaches.
  arXiv  Detail & Related papers  (2025-09-22T11:30:39Z)
• Introducing MeMo: A Multimodal Dataset for Memory Modelling in Multiparty Conversations [1.8896253910986929]
  MeMo corpus is the first dataset annotated with participants' memory retention reports. It integrates validated behavioural and perceptual measures, audio, video, and multimodal annotations. This paper aims to pave the way for future research in conversational memory modelling for intelligent system development.
  arXiv  Detail & Related papers  (2024-09-07T16:09:36Z)
• Hello Again! LLM-powered Personalized Agent for Long-term Dialogue [63.65128176360345]
  We introduce a model-agnostic framework, the Long-term Dialogue Agent (LD-Agent)<n>It incorporates three independently tunable modules dedicated to event perception, persona extraction, and response generation.<n>The effectiveness, generality, and cross-domain capabilities of LD-Agent are empirically demonstrated.
  arXiv  Detail & Related papers  (2024-06-09T21:58:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.