A Grounded Memory System For Smart Personal Assistants

• URL: http://arxiv.org/abs/2505.06328v1
• Date: Fri, 09 May 2025 10:08:22 GMT
• Title: A Grounded Memory System For Smart Personal Assistants
• Authors: Felix Ocker, Jörg Deigmöller, Pavel Smirnov, Julian Eggert,
• Abstract summary: A wide variety of agentic AI applications - ranging from cognitive assistants for dementia patients to robotics - demand a robust memory system grounded in reality.<n>We propose such a memory system consisting of three components.<n>First, we combine Vision Language Models for image captioning and entity disambiguation with Large Language Models for consistent information extraction during perception.<n>Second, the extracted information is represented in a memory consisting of a knowledge graph enhanced by vector embeddings to efficiently manage relational information.<n>Third, we combine semantic search and graph query generation for question answering via Retrieval Augmented Generation.
• Score: 1.5267291767316298
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A wide variety of agentic AI applications - ranging from cognitive assistants for dementia patients to robotics - demand a robust memory system grounded in reality. In this paper, we propose such a memory system consisting of three components. First, we combine Vision Language Models for image captioning and entity disambiguation with Large Language Models for consistent information extraction during perception. Second, the extracted information is represented in a memory consisting of a knowledge graph enhanced by vector embeddings to efficiently manage relational information. Third, we combine semantic search and graph query generation for question answering via Retrieval Augmented Generation. We illustrate the system's working and potential using a real-world example.

Related papers

• Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions [55.19217798774033]
  Memory is a fundamental component of AI systems, underpinning large language models (LLMs)-based agents.<n>In this survey, we first categorize memory representations into parametric and contextual forms.<n>We then introduce six fundamental memory operations: Consolidation, Updating, Indexing, Forgetting, Retrieval, and Compression.
  arXiv  Detail & Related papers  (2025-05-01T17:31:33Z)
• From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs [34.361000444808454]
  Memory is the process of encoding, storing, and retrieving information.<n>In the era of large language models (LLMs), memory refers to the ability of an AI system to retain, recall, and use information from past interactions to improve future responses and interactions.
  arXiv  Detail & Related papers  (2025-04-22T15:05:04Z)
• DEMENTIA-PLAN: An Agent-Based Framework for Multi-Knowledge Graph Retrieval-Augmented Generation in Dementia Care [3.9891568002886766]
  We propose DEMENTIA-PLAN, an innovative retrieval-augmented generation framework.<n>Our model employs a multiple knowledge graph architecture, integrating various dimensional knowledge representations.<n>Our notable innovation is the self-reflection planning agent, which coordinates knowledge retrieval and semantic integration.
  arXiv  Detail & Related papers  (2025-03-26T19:34:04Z)
• 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning [65.40458559619303]
  We propose 3D-Mem, a novel 3D scene memory framework for embodied agents.<n>3D-Mem employs informative multi-view images, termed Memory Snapshots, to represent the scene.<n>It further integrates frontier-based exploration by introducing Frontier Snapshots-glimpses of unexplored areas-enabling agents to make informed decisions.
  arXiv  Detail & Related papers  (2024-11-23T09:57:43Z)
• Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation [69.01029651113386]
  Embodied-RAG is a framework that enhances the model of an embodied agent with a non-parametric memory system.<n>At its core, Embodied-RAG's memory is structured as a semantic forest, storing language descriptions at varying levels of detail.<n>We demonstrate that Embodied-RAG effectively bridges RAG to the robotics domain, successfully handling over 250 explanation and navigation queries.
  arXiv  Detail & Related papers  (2024-09-26T21:44:11Z)
• Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction [8.63068449082585]
  Decoding non-invasive brain recordings is pivotal for advancing our understanding of human cognition. Our framework integrates 3D brain structures with visual semantics using a Vision Transformer 3D. We have enhanced the fMRI dataset with diverse fMRI-image-related textual data to support multimodal large model development.
  arXiv  Detail & Related papers  (2024-04-30T10:41:23Z)
• Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning [64.1316997189396]
  We present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets.
  arXiv  Detail & Related papers  (2024-03-21T17:58:56Z)
• A Framework for Inference Inspired by Human Memory Mechanisms [9.408704431898279]
  We propose a PMI framework that consists of perception, memory and inference components. The memory module comprises working and long-term memory, with the latter endowed with a higher-order structure to retain extensive and complex relational knowledge and experience. We apply our PMI to improve prevailing Transformers and CNN models on question-answering tasks like bAbI-20k and Sort-of-CLEVR datasets.
  arXiv  Detail & Related papers  (2023-10-01T08:12:55Z)
• Improving Image Recognition by Retrieving from Web-Scale Image-Text Data [68.63453336523318]
  We introduce an attention-based memory module, which learns the importance of each retrieved example from the memory. Compared to existing approaches, our method removes the influence of the irrelevant retrieved examples, and retains those that are beneficial to the input query. We show that it achieves state-of-the-art accuracies in ImageNet-LT, Places-LT and Webvision datasets.
  arXiv  Detail & Related papers  (2023-04-11T12:12:05Z)
• A Machine with Short-Term, Episodic, and Semantic Memory Systems [9.42475956340287]
  Inspired by the cognitive science theory of the explicit human memory systems, we have modeled an agent with short-term, episodic, and semantic memory systems. Our experiments indicate that an agent with human-like memory systems can outperform an agent without this memory structure in the environment.
  arXiv  Detail & Related papers  (2022-12-05T08:34:23Z)
• Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery [84.73764603474413]
  We propose a novel online approach of multi-modal graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset.
  arXiv  Detail & Related papers  (2020-11-03T11:00:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.