EgoTrigger: Toward Audio-Driven Image Capture for Human Memory Enhancement in All-Day Energy-Efficient Smart Glasses

• URL: http://arxiv.org/abs/2508.01915v1
• Date: Sun, 03 Aug 2025 20:51:23 GMT
• Title: EgoTrigger: Toward Audio-Driven Image Capture for Human Memory Enhancement in All-Day Energy-Efficient Smart Glasses
• Abstract summary: EgoTrigger uses audio cues from the microphone to selectively activate power-intensive cameras.<n>EgoTrigger can use 54% fewer frames on average, significantly saving energy in both power-hungry sensing components.<n>We believe this context-aware triggering strategy represents a promising direction for enabling energy-efficient, functional smart glasses capable of all-day use.
• Score: 9.926912962527958
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: All-day smart glasses are likely to emerge as platforms capable of continuous contextual sensing, uniquely positioning them for unprecedented assistance in our daily lives. Integrating the multi-modal AI agents required for human memory enhancement while performing continuous sensing, however, presents a major energy efficiency challenge for all-day usage. Achieving this balance requires intelligent, context-aware sensor management. Our approach, EgoTrigger, leverages audio cues from the microphone to selectively activate power-intensive cameras, enabling efficient sensing while preserving substantial utility for human memory enhancement. EgoTrigger uses a lightweight audio model (YAMNet) and a custom classification head to trigger image capture from hand-object interaction (HOI) audio cues, such as the sound of a drawer opening or a medication bottle being opened. In addition to evaluating on the QA-Ego4D dataset, we introduce and evaluate on the Human Memory Enhancement Question-Answer (HME-QA) dataset. Our dataset contains 340 human-annotated first-person QA pairs from full-length Ego4D videos that were curated to ensure that they contained audio, focusing on HOI moments critical for contextual understanding and memory. Our results show EgoTrigger can use 54% fewer frames on average, significantly saving energy in both power-hungry sensing components (e.g., cameras) and downstream operations (e.g., wireless transmission), while achieving comparable performance on datasets for an episodic memory task. We believe this context-aware triggering strategy represents a promising direction for enabling energy-efficient, functional smart glasses capable of all-day use -- supporting applications like helping users recall where they placed their keys or information about their routine activities (e.g., taking medications).

Related papers

• LightMem-Ego: Your AI Memory for Everyday Life [15.591017916133309]
  We present LightMem-Ego, a lightweight streaming multimodal memory system for everyday-life assistance.<n>The system continuously captures egocentric visual and audio streams, aligns them on a shared timeline, and organizes them into a hierarchical memory.<n>Given a user query, LightMem-Ego dynamically routes retrieval to the appropriate memory level and generates answers grounded in multimodal evidence.
  arXiv  Detail & Related papers  (2026-07-13T12:40:17Z)
• S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval [38.608941713774186]
  We introduce S-EMBER (Streaming Egocentric Memory Benchmark for Episodic Retrieval), a large-scale benchmark comprising 3,141 videos totaling 388 hours of organic activity captured via Ray-Ban Meta smart glasses.<n>S-EMBER formalizes grounded streaming episodic retrieval, a paradigm shift from global offline search to causal, active recall triggered by visual events in a continuous stream.
  arXiv  Detail & Related papers  (2026-07-02T18:29:20Z)
• Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams [58.77207336324662]
  UCS-Bench is a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions.<n>We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations.
  arXiv  Detail & Related papers  (2026-06-13T08:50:49Z)
• SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory [14.96864764261795]
  SuperMemory-VQA is a dataset for evaluating AI assistants on practical, long-horizon memory tasks.<n>It contains 52.9 hours of everyday activities recorded with AI glasses, including synchronized RGB video, audio transcription, eye gaze, IMU, and SLAM trajectories.<n>We construct grounded 4,853 question-answer pairs that span object and location memory, intent recall, visual scene recall, timeline reconstruction, conversational memory, and in-context retrieval.
  arXiv  Detail & Related papers  (2026-05-30T17:53:10Z)
• Exploring Audio Hallucination in Egocentric Video Understanding [66.1760617001607]
  State-of-the-art large audio-visual language models (AV-LLMs) can generate multimodal descriptions.<n>We show in this work that they are prone to audio hallucinations, often inferring sounds from visual cues that are visible but not heard.
  arXiv  Detail & Related papers  (2026-04-26T20:06:58Z)
• From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents [78.30630000529133]
  We propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory.<n> MM-Mem memory structures hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic.<n>Experiments confirm the effectiveness of MM-Mem on both offline and streaming tasks.
  arXiv  Detail & Related papers  (2026-03-02T05:12:45Z)
• O-Mem: Omni Memory System for Personalized, Long Horizon, Self-Evolving Agents [60.1848551962911]
  O-Mem is a novel memory framework based on active user profiling.<n>O-Mem supports hierarchical retrieval of persona attributes and topic-related context.
  arXiv  Detail & Related papers  (2025-11-17T16:55:19Z)
• TeleEgo: Benchmarking Egocentric AI Assistants in the Wild [55.53194302888826]
  Egocentric AI assistants in real-world settings must process multi-modal inputs (video, audio, text)<n>We introduce textbfTeleEgo, a long-duration, streaming, omni-modal benchmark for evaluating egocentric AI assistants.<n>The dataset features over 14 hours per participant of synchronized egocentric video, audio, and text across four domains.
  arXiv  Detail & Related papers  (2025-10-28T01:24:24Z)
• LightMem: Lightweight and Efficient Memory-Augmented Generation [72.21680105265824]
  We introduce a new memory system called LightMem, which strikes a balance between the performance and efficiency of memory systems.<n>Inspired by the Atkinson-Shiffrin model of human memory, LightMem organizes memory into three complementary stages.<n>Experiments on LongMemEval with GPT and Qwen backbones show that LightMem outperforms strong baselines in accuracy (up to 10.9% gains) while reducing token usage by up to 117x.
  arXiv  Detail & Related papers  (2025-10-21T17:58:17Z)
• AHA - Predicting What Matters Next: Online Highlight Detection Without Looking Ahead [4.55107996328448]
  Aha is an autoregressive highlight detection framework that predicts relevance of each video frame against a task described in natural language.<n>Aha achieves state-of-the-art (SOTA) performance on highlight detection benchmarks.<n>We explore Aha's potential for real-world robotics applications given a task-oriented natural language input and a continuous, robot-centric video.
  arXiv  Detail & Related papers  (2025-09-19T21:03:00Z)
• Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input [62.51283548975632]
  This work focuses on tracking and understanding human motion using consumer wearable devices, such as VR/AR headsets, smart glasses, cellphones, and smartwatches.<n>We present Ego4o (o for omni), a new framework for simultaneous human motion capture and understanding from multi-modal egocentric inputs.
  arXiv  Detail & Related papers  (2025-04-11T11:18:57Z)
• EgoLife: Towards Egocentric Life Assistant [60.51196061794498]
  We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses.<n>We conduct a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references.<n>This effort resulted in the EgoLife dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation.<n>We introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide
  arXiv  Detail & Related papers  (2025-03-05T18:54:16Z)
• Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding [21.619878862869754]
  Embodied VideoAgent constructs scene memory from both egocentric video and embodied sensory inputs.<n>We have demonstrated its potential in various embodied AI tasks including generating embodied interactions and perception for robot manipulation.
  arXiv  Detail & Related papers  (2024-12-31T09:22:38Z)
• Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory [17.305576093380168]
  We introduce Visual Online 2D (OVQ2D), a task where models process streams online, observing each frame only once, and retrieve object localizations using a compact memory instead of full full video history.<n>We demonstrate ESOM videos superiority over other online approaches though OVQ2D remains challenging, with top performance at only 4% success.
  arXiv  Detail & Related papers  (2024-11-25T21:07:25Z)
• Predictive Temporal Attention on Event-based Video Stream for Energy-efficient Situation Awareness [5.541459858161597]
  We propose a temporal attention mechanism to throttle the camera output and pay attention to it only when the visual events cannot be well predicted. We demonstrate that the predictive attention can reduce 46.7% of data communication between the camera and the processor and reduce 43.8% activities in the processor.
  arXiv  Detail & Related papers  (2024-02-14T04:34:48Z)
• MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World [55.878173953175356]
  We propose MultiPLY, a multisensory embodied large language model. We first collect Multisensory Universe, a large-scale multisensory interaction dataset comprising 500k data. We demonstrate that MultiPLY outperforms baselines by a large margin through a diverse set of embodied tasks.
  arXiv  Detail & Related papers  (2024-01-16T18:59:45Z)
• Encode-Store-Retrieve: Augmenting Human Memory through Language-Encoded Egocentric Perception [19.627636189321393]
  A promising avenue for memory augmentation is through the use of augmented reality head-mounted displays to capture and preserve egocentric videos. The current technology lacks the capability to encode and store such large amounts of data efficiently. We propose a memory augmentation agent that involves leveraging natural language encoding for video data and storing them in a vector database.
  arXiv  Detail & Related papers  (2023-08-10T18:43:44Z)
• Egocentric Audio-Visual Object Localization [51.434212424829525]
  We propose a geometry-aware temporal aggregation module to handle the egomotion explicitly. The effect of egomotion is mitigated by estimating the temporal geometry transformation and exploiting it to update visual representations. It improves cross-modal localization robustness by disentangling visually-indicated audio representation.
  arXiv  Detail & Related papers  (2023-03-23T17:43:11Z)
• Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning [62.83590925557013]
  We learn a set of challenging partially-observed manipulation tasks from visual and audio inputs. Our proposed system learns these tasks by combining offline imitation learning from tele-operated demonstrations and online finetuning. In a set of simulated tasks, we find that our system benefits from using audio, and that by using online interventions we are able to improve the success rate of offline imitation learning by 20%.
  arXiv  Detail & Related papers  (2022-05-30T04:52:58Z)
• Episodic Memory Question Answering [55.83870351196461]
  We envision a scenario wherein the human communicates with an AI agent powering an augmented reality device by asking questions. In order to succeed, the ego AI assistant must construct semantically rich and efficient scene memories. We introduce a new task - Episodic Memory Question Answering (EMQA) We show that our choice of episodic scene memory outperforms naive, off-the-centric solutions for the task as well as a host of very competitive baselines.
  arXiv  Detail & Related papers  (2022-05-03T17:28:43Z)
• Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention [0.9668407688201357]
  We propose a new task of sensor-augmented egocentric-video captioning. We use wearable-sensor data as auxiliary information to mitigate the inherent problems in egocentric vision.
  arXiv  Detail & Related papers  (2021-09-07T09:22:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.