Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding

• URL: http://arxiv.org/abs/2608.07663v1
• Date: Fri, 07 Aug 2026 18:00:02 GMT
• Title: Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding
• Abstract summary: We propose MERIT, a simple yet effective framework for ultra-long video understanding.<n>We formulate an episodic multi-key representation that enables precise retrieval of fine-grained memories.<n>MeRIT achieves state-of-the-art performance across three long-video benchmarks.
• Score: 41.45143308235023
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When videos extend from hours to days, directly processing them end-to-end becomes impractical for current Multi-modal Large Language Models (MLLMs). This ultra-long setting necessitates a two-stage paradigm: query-agnostic memory construction followed by retrieval-based inference. Prior work invests in complex memory construction to pre-model high-level relations in videos, despite not knowing the downstream query at build time. We instead prioritize high-recall retrievability during memory building, and defer query-specific, high-level relation composition to inference time. To this end, we propose MERIT(Multi-key Episodic Retrieval with Inference-time Temporal expansion), a simple yet effective agentic framework for ultra-long video understanding. First, we formulate an episodic multi-key representation that enables precise retrieval of fine-grained memories through a simple key-matching mechanism. Second, we introduce a neighbor filtering mechanism to capture broader semantic context without the massive computational overhead of global memory construction. This is achieved by expanding the temporal scope exclusively around the retrieved segments at inference time. By leveraging simple key-matching with this on-demand temporal expansion, MERIT achieves state-of-the-art performance across three long-video benchmarks: EgoLifeQA, LVBench, and Video-MME (Long).

Related papers

• Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning [77.7531245219403]
  textscHomer is a Hierarchical Online Memory Exploration and Reasoning framework.<n>textscHomer's memory mirrors the multi-scale structure of long videos, ranging from raw perception, to recurring entities, to events connected by explicit temporal and causal relations.<n>textscHomer outperforms the previous best agent method by $+5.5$, $+10.8$, and $+4.4$ points on M3-Bench-robot, M3-Bench-web, and Video-MME-Long.
  arXiv  Detail & Related papers  (2026-07-01T05:53:09Z)
• Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding [47.48582887762574]
  ReMem is a temporal granularity-adaptive selection framework for training-free LongVideoQA.<n>By explicitly preserving temporal information with memory mechanisms, ReMem suppresses redundancy and empowers MLLMs to perform robust multi-granular video reasoning.
  arXiv  Detail & Related papers  (2026-06-30T10:55:56Z)
• WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning [66.24870234484668]
  We introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories.<n>WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks.
  arXiv  Detail & Related papers  (2025-12-02T05:14:52Z)
• GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memory [59.869552603264076]
  We introduce GCAgent, a novel Global-Context-Aware Agent framework that achieves comprehensive long-video understanding.<n>Our core innovation is the Schematic and Narrative Episodic Memory, which structurally models events and their causal and temporal relations into a concise, organized context.<n>Experiments confirm that GCAgent significantly enhances long-video understanding, achieving up to 23.5% accuracy improvement on the Video-MME Long split over a strong MLLM baseline.
  arXiv  Detail & Related papers  (2025-11-15T04:29:00Z)
• Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignment [0.0]
  We propose UMaT, a framework that unifies visual and auditory inputs as structured text for large language models.<n>It significantly improves state-of-the-art Long Video Question Answering accuracy.
  arXiv  Detail & Related papers  (2025-03-12T05:28:24Z)
• InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions [104.90258030688256]
  This project introduces disentangled streaming perception, reasoning, and memory mechanisms, enabling real-time interaction with streaming video and audio input.<n>This project simulates human-like cognition, enabling multimodal large language models to provide continuous and adaptive service over time.
  arXiv  Detail & Related papers  (2024-12-12T18:58:30Z)
• ReWind: Understanding Long Videos with Instructed Learnable Memory [8.002949551539297]
  Vision-Language Models (VLMs) are crucial for applications requiring integrated understanding textual and visual information.<n>We introduce ReWind, a novel memory-based VLM designed for efficient long video understanding while preserving temporal fidelity.<n>We empirically demonstrate ReWind's superior performance in visual question answering (VQA) and temporal grounding tasks, surpassing previous methods on long video benchmarks.
  arXiv  Detail & Related papers  (2024-11-23T13:23:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.