Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge

• URL: http://arxiv.org/abs/2602.22455v1
• Date: Wed, 25 Feb 2026 22:30:56 GMT
• Title: Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge
• Abstract summary: We investigate the feasibility of using Multimodal Large Language Models (MLLMs) for real-time online episodic memory question answering.<n>Experiments on the QAEgo4D-Closed benchmark analyze the performance of Multimodal Large Language Models (MLLMs) within strict resource boundaries.
• Score: 9.701124246177661
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate the feasibility of using Multimodal Large Language Models (MLLMs) for real-time online episodic memory question answering. While cloud offloading is common, it raises privacy and latency concerns for wearable assistants, hence we investigate implementation on the edge. We integrated streaming constraints into our question answering pipeline, which is structured into two asynchronous threads: a Descriptor Thread that continuously converts video into a lightweight textual memory, and a Question Answering (QA) Thread that reasons over the textual memory to answer queries. Experiments on the QAEgo4D-Closed benchmark analyze the performance of Multimodal Large Language Models (MLLMs) within strict resource boundaries, showing promising results also when compared to clound-based solutions. Specifically, an end-to-end configuration running on a consumer-grade 8GB GPU achieves 51.76% accuracy with a Time-To-First-Token (TTFT) of 0.41s. Scaling to a local enterprise-grade server yields 54.40% accuracy with a TTFT of 0.88s. In comparison, a cloud-based solution obtains an accuracy of 56.00%. These competitive results highlight the potential of edge-based solutions for privacy-preserving episodic memory retrieval.

Related papers

• MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models [50.25006399944962]
  Memory is essential for large vision-language models to handle long, multimodal interactions.<n>MEMLENS is a benchmark for memory in multimodal multi-session conversations.<n>We evaluate 27 LVLMs and 7 memory-augmented agents.
  arXiv  Detail & Related papers  (2026-05-14T14:41:17Z)
• VSAS-BENCH: Real-Time Evaluation of Visual Streaming Assistant Models [39.78979236902648]
  Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames.<n>We propose VSAS-Bench, a new framework and benchmark for Visual Streaming Assistants.
  arXiv  Detail & Related papers  (2026-04-08T22:31:20Z)
• A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos [76.98722001848493]
  LongShOTBench is a diagnostic benchmark for long-form multimodal video understanding.<n>It includes open-ended, intent-driven questions; single- and multi-turn dialogues; and tasks requiring multimodal reasoning and agentic tool use.<n>LongShOTAgent is an agentic system that analyzes long videos via preprocessing, search, and iterative refinement.
  arXiv  Detail & Related papers  (2025-12-18T18:59:27Z)
• video-SALMONN S: Streaming Audio-Visual LLMs Beyond Length Limits via Memory [51.03819128505358]
  Video-SALMONN S is first to process 3-hour videos at 1 FPS and 360p resolution under a fixed memory budget.<n>A test-time-training memory module continually updates token representations to capture long-range dependencies.<n>A prompt-dependent memory reader retrieves context-relevant content from fixed-size memory.
  arXiv  Detail & Related papers  (2025-10-13T08:20:15Z)
• How Far Can Off-the-Shelf Multimodal Large Language Models Go in Online Episodic Memory Question Answering? [11.937961684293672]
  We investigate whether off-the-shelf Multimodal Large Language Models (MLLMs) can tackle Online Episodic-Memory Video Question Answering (OEM-VQA) without additional training.<n>Our pipeline converts a streaming egocentric video into a lightweight textual memory, only a few kilobytes per minute, via an MLLM descriptor module, and answers multiple-choice questions by querying this memory with an LLM reasoner module.
  arXiv  Detail & Related papers  (2025-06-19T16:35:49Z)
• PentaRAG: Large-Scale Intelligent Knowledge Retrieval for Enterprise LLM Applications [5.4838799162708245]
  We present PentaRAG, a five-layer module that routes each query through two instant caches.<n>We show that PentaRAG cuts average GPU time to 0.248 seconds per query, roughly half that of a naive RAG baseline.<n>Results demonstrate that a layered routing strategy can deliver freshness, speed, and efficiency simultaneously in production-grade RAG systems.
  arXiv  Detail & Related papers  (2025-06-18T07:54:53Z)
• MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks [67.31276358668424]
  We introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer.<n> AVHaystacks is an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task.<n>We propose a model-agnostic, multi-agent framework to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystack
  arXiv  Detail & Related papers  (2025-06-08T06:34:29Z)
• SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding [56.78088668917983]
  We introduce SVBench, a pioneering benchmark with temporal multi-turn question-answering chains.<n>We design a semi-automated annotation pipeline to obtain 49,979 Question-Answer (QA) pairs of 1,353 streaming videos.<n>Our experimental results, obtained from 14 models in dialogue and streaming evaluations, reveal that while the closed-source GPT-4o outperforms others, most open-source LVLMs struggle with long-context streaming video understanding.
  arXiv  Detail & Related papers  (2025-02-15T14:29:44Z)
• LiVOS: Light Video Object Segmentation with Gated Linear Matching [116.58237547253935]
  LiVOS is a lightweight memory network that employs linear matching via linear attention. For longer and higher-resolution videos, it matched STM-based methods with 53% less GPU memory and supports 4096p inference on a 32G consumer-grade GPU.
  arXiv  Detail & Related papers  (2024-11-05T05:36:17Z)
• TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices [36.714057078457195]
  We present TPI-LLM, a compute- and memory-efficient tensor parallel inference system for 70B-scale models. TPI-LLM keeps sensitive raw data local in the users' devices and introduces a sliding window memory scheduler. We show that TPI-LLM demonstrated over 80% less time-to-first-token and token latency compared to Accelerate.
  arXiv  Detail & Related papers  (2024-10-01T09:18:56Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• Recurrent Context Compression: Efficiently Expanding the Context Window of LLM [22.595457889113668]
  This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of Transformer-based large language models (LLMs) We validated our approach on multiple tasks, achieving a compression rate of up to 32x on text reconstruction tasks with a BLEU4 score close to 0.95, and nearly 100% accuracy on a passkey retrieval task with a sequence length of 1M.
  arXiv  Detail & Related papers  (2024-06-10T08:50:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.