OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering

• URL: http://arxiv.org/abs/2409.08250v2
• Date: Fri, 21 Feb 2025 02:46:25 GMT
• Title: OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering
• Abstract summary: We introduce OmniQuery, a novel system that is able to answer complex personal memory-related questions.<n>Given a question, OmniQuery retrieves relevant augmented memories and uses a large language model (LLM) to generate answers with references.<n>In human evaluations, we show the effectiveness of OmniQuery with an accuracy of 71.5%, outperforming a conventional RAG system by winning or tying for 74.5% of the time.
• Score: 3.8233569758620063
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: People often capture memories through photos, screenshots, and videos. While existing AI-based tools enable querying this data using natural language, they only support retrieving individual pieces of information like certain objects in photos, and struggle with answering more complex queries that involve interpreting interconnected memories like sequential events. We conducted a one-month diary study to collect realistic user queries and generated a taxonomy of necessary contextual information for integrating with captured memories. We then introduce OmniQuery, a novel system that is able to answer complex personal memory-related questions that require extracting and inferring contextual information. OmniQuery augments individual captured memories through integrating scattered contextual information from multiple interconnected memories. Given a question, OmniQuery retrieves relevant augmented memories and uses a large language model (LLM) to generate answers with references. In human evaluations, we show the effectiveness of OmniQuery with an accuracy of 71.5%, outperforming a conventional RAG system by winning or tying for 74.5% of the time.

Related papers

• Personal AI Agent for Camera Roll VQA [61.305160563081614]
  We study the personal camera roll visual question answering setting.<n>In this setting, a conversational AI assistant can access a user's personal camera roll and retrieve relevant photos to answer queries.<n>To support this, we collect and manually annotate questions that mimic real-world usage.
  arXiv  Detail & Related papers  (2026-06-03T17:59:30Z)
• SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory [14.96864764261795]
  SuperMemory-VQA is a dataset for evaluating AI assistants on practical, long-horizon memory tasks.<n>It contains 52.9 hours of everyday activities recorded with AI glasses, including synchronized RGB video, audio transcription, eye gaze, IMU, and SLAM trajectories.<n>We construct grounded 4,853 question-answer pairs that span object and location memory, intent recall, visual scene recall, timeline reconstruction, conversational memory, and in-context retrieval.
  arXiv  Detail & Related papers  (2026-05-30T17:53:10Z)
• Memory-Augmented Query Intent Understanding for Efficient Chat-based Image Retrieval [74.93546115702155]
  Chat-based image retrieval allows the human-interactive system to iteratively clarify and refine user intent through multi-round dialogue.<n>The key challenge in this task lies in dynamically understanding and updating the user's query intent across dialogue rounds.<n>This paper proposes a novel and efficient memory-based user intent updating framework for the chat-based image retrieval task.
  arXiv  Detail & Related papers  (2026-05-17T10:17:41Z)
• APEX-MEM: Agentic Semi-Structured Memory with Temporal Reasoning for Long-Term Conversational AI [8.933188286750795]
  APEX-MEM is a conversational memory system that combines three key innovations.<n>Property graphs structure conversations as temporally grounded events in an entity-centric framework.<n>Retrieval agent understands and resolves conflicting or evolving information at query time, producing a compact and contextually relevant memory summary.
  arXiv  Detail & Related papers  (2026-04-15T19:25:02Z)
• OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering [12.963349283282447]
  We propose OmniRAG-Agent, an agentic omnimodal QA method for budgeted long audio-video reasoning.<n>It builds an image-audio retrieval-augmented generation module that lets an OmniLLM fetch short, relevant frames and audio snippets from external banks.<n>It uses an agent loop that plans, calls tools across turns, and merges retrieved evidence to answer complex queries.
  arXiv  Detail & Related papers  (2026-02-03T16:28:24Z)
• Omni-Referring Image Segmentation [106.67482846604877]
  We propose a novel task Omni-Referring Image termed ( OmniRIS) towards highly generalized image segmentation.<n>Compared with existing unimodally conditioned segmentation tasks, such as RIS and visual RIS, OmniRIS supports the input of text instructions and reference images with masks, boxes or scribbles as omni-prompts.<n>We rigorously design and construct a large dataset termed OmniRef, which consists of 186,939 omni-prompts for 30,956 images.
  arXiv  Detail & Related papers  (2025-12-07T14:32:34Z)
• WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning [66.24870234484668]
  We introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories.<n>WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks.
  arXiv  Detail & Related papers  (2025-12-02T05:14:52Z)
• Evaluating Long-Term Memory for Long-Context Question Answering [100.1267054069757]
  We present a systematic evaluation of memory-augmented methods using LoCoMo, a benchmark of synthetic long-context dialogues annotated for question-answering tasks.<n>Our findings show that memory-augmented approaches reduce token usage by over 90% while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2025-10-27T18:03:50Z)
• EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models [40.712154207072594]
  EgoMem is a lifelong memory agent tailored for realtime omnimodal streams.<n>The system achieves fact-consistency scores above 87% in real-time personalized dialogs.
  arXiv  Detail & Related papers  (2025-09-15T13:33:29Z)
• Towards Multi-Granularity Memory Association and Selection for Long-Term Conversational Agents [73.77930932005354]
  We propose MemGAS, a framework that enhances memory consolidation by constructing multi-granularity association, adaptive selection, and retrieval.<n>MemGAS is based on multi-granularity memory units and employs Gaussian Mixture Models to cluster and associate new memories with historical ones.<n>Experiments on four long-term memory benchmarks demonstrate that MemGAS outperforms state-of-the-art methods on both question answer and retrieval tasks.
  arXiv  Detail & Related papers  (2025-05-26T06:13:07Z)
• A Grounded Memory System For Smart Personal Assistants [1.5267291767316298]
  A wide variety of agentic AI applications - ranging from cognitive assistants for dementia patients to robotics - demand a robust memory system grounded in reality.<n>We propose such a memory system consisting of three components.<n>First, we combine Vision Language Models for image captioning and entity disambiguation with Large Language Models for consistent information extraction during perception.<n>Second, the extracted information is represented in a memory consisting of a knowledge graph enhanced by vector embeddings to efficiently manage relational information.<n>Third, we combine semantic search and graph query generation for question answering via Retrieval Augmented Generation.
  arXiv  Detail & Related papers  (2025-05-09T10:08:22Z)
• Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions [55.19217798774033]
  Memory is a fundamental component of AI systems, underpinning large language models (LLMs) based agents. We introduce six fundamental memory operations: Consolidation, Updating, Indexing, Forgetting, Retrieval, and Compression. This survey provides a structured and dynamic perspective on research, benchmark datasets, and tools related to memory in AI.
  arXiv  Detail & Related papers  (2025-05-01T17:31:33Z)
• Memorization and Knowledge Injection in Gated LLMs [8.305942415868042]
  Large Language Models (LLMs) currently struggle to sequentially add new memories and integrate new knowledge. Memory Embedded in Gated LLMs (MEGa) injects event memories directly into the weights of LLMs. During inference, a gating mechanism activates relevant memory weights by matching query embeddings to stored memory embeddings.
  arXiv  Detail & Related papers  (2025-04-30T00:28:32Z)
• OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup [50.70494796172493]
  We introduce Omni-modal Sound Separation (OmniSep), a novel framework capable of isolating clean soundtracks based on omni-modal queries. We introduce the Query-Mixup strategy, which blends query features from different modalities during training. We further enhance this flexibility by allowing queries to influence sound separation positively or negatively, facilitating the retention or removal of specific sounds.
  arXiv  Detail & Related papers  (2024-10-28T17:58:15Z)
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory [68.97819665784442]
  This paper introduces LongMemEval, a benchmark designed to evaluate five core long-term memory abilities of chat assistants. LongMemEval presents a significant challenge to existing long-term memory systems. We present a unified framework that breaks down the long-term memory design into four design choices.
  arXiv  Detail & Related papers  (2024-10-14T17:59:44Z)
• PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering [27.815507347725344]
  This research introduces PerLTQA, an innovative QA dataset that combines semantic and episodic memories. PerLTQA features two types of memory and a benchmark of 8,593 questions for 30 characters. We propose a novel framework for memory integration and generation, consisting of three main components: Memory Classification, Memory Retrieval, and Memory Synthesis.
  arXiv  Detail & Related papers  (2024-02-26T04:09:53Z)
• Multi-Clue Reasoning with Memory Augmentation for Knowledge-based Visual Question Answering [32.21000330743921]
  We propose a novel framework that endows the model with capabilities of answering more general questions. Specifically, a well-defined detector is adopted to predict image-question related relation phrases. The optimal answer is predicted by choosing the supporting fact with the highest score.
  arXiv  Detail & Related papers  (2023-12-20T02:35:18Z)
• UniMC: A Unified Framework for Long-Term Memory Conversation via Relevance Representation Learning [15.313416157905685]
  We propose a Unified framework for Long-term Memory Conversations (UniMC) We decompose the main task into three subtasks based on probability graphs. Each subtask involves learning a representation for calculating the relevance between the query and memory.
  arXiv  Detail & Related papers  (2023-06-18T12:30:50Z)
• End-to-end Knowledge Retrieval with Multi-modal Queries [50.01264794081951]
  ReMuQ requires a system to retrieve knowledge from a large corpus by integrating contents from both text and image queries. We introduce a retriever model ReViz'' that can directly process input text and images to retrieve relevant knowledge in an end-to-end fashion. We demonstrate superior performance in retrieval on two datasets under zero-shot settings.
  arXiv  Detail & Related papers  (2023-06-01T08:04:12Z)
• Chatting Makes Perfect: Chat-based Image Retrieval [25.452015862927766]
  ChatIR is a chat-based image retrieval system that engages in a conversation with the user to elicit information. Large Language Models are used to generate follow-up questions to an initial image description. Our system is capable of retrieving the target image from a pool of 50K images with over 78% success rate after 5 dialogue rounds.
  arXiv  Detail & Related papers  (2023-05-31T17:38:08Z)
• RET-LLM: Towards a General Read-Write Memory for Large Language Models [53.288356721954514]
  RET-LLM is a novel framework that equips large language models with a general write-read memory unit. Inspired by Davidsonian semantics theory, we extract and save knowledge in the form of triplets. Our framework exhibits robust performance in handling temporal-based question answering tasks.
  arXiv  Detail & Related papers  (2023-05-23T17:53:38Z)
• REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory [119.98011559193574]
  We propose an end-to-end Retrieval-Augmented Visual Language Model (REVEAL) It learns to encode world knowledge into a large-scale memory, and to retrieve from it to answer knowledge-intensive queries. A key novelty in our approach is that the memory, encoder, retriever and generator are all pre-trained end-to-end on a massive amount of data.
  arXiv  Detail & Related papers  (2022-12-10T06:17:56Z)
• Navigating Connected Memories with a Task-oriented Dialog System [13.117491508194242]
  We propose dialogs for connected memories as a powerful tool to empower users to search their media collection through a multi-turn, interactive conversation. We use a new task-oriented dialog dataset COMET, which contains $11.5k$ user->assistant dialogs (totaling $103k$ utterances) grounded in simulated personal memory graphs. We analyze COMET, formulate four main tasks to benchmark meaningful progress, and adopt state-of-the-art language models as strong baselines.
  arXiv  Detail & Related papers  (2022-11-15T19:31:57Z)
• MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text [58.655375327681774]
  We propose the first Multimodal Retrieval-Augmented Transformer (MuRAG) MuRAG accesses an external non-parametric multimodal memory to augment language generation. Our results show that MuRAG achieves state-of-the-art accuracy, outperforming existing models by 10-20% absolute on both datasets.
  arXiv  Detail & Related papers  (2022-10-06T13:58:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.