Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos
- URL: http://arxiv.org/abs/2607.11523v1
- Date: Mon, 13 Jul 2026 13:09:45 GMT
- Title: Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos
- Abstract summary: Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive.<n>We present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity.<n>EgoMemo is a training-free, memory-augmented agent that maintains three complementary memory representations.
- Score: 74.71281081304846
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at \href{https://sitonggong.github.io/EgoServe-page/}{Vinci2}.
Related papers
- EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports [5.919102792488079]
We introduce EgoArgus, a human-annotated dataset for evaluating egocentric assistants on understanding and decision tasks.<n>Our results demonstrate that it is still challenging for current VLMs as reliable egocentric assistants.
arXiv Detail & Related papers (2026-08-26T09:09:30Z) - EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory [10.606058056542734]
EgoCITE is a long-horizon agentic memory framework for egocentric QA.<n>EgoIndex organizes complementary action, activity, utterance, and conversation representations into searchable memory indices.<n>EgoRetrv combines semantic search with question-conditioned temporal relevance scoring and curation of retrieved evidence.
arXiv Detail & Related papers (2026-08-12T22:20:29Z) - EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning [47.853306116245484]
EgoIntrospect is the first egocentric dataset captured in user-driven scenarios with self-annotations.<n>It consists of 180 hours of recordings from 60 subjects, with an average recording duration of 3 hours per subject.<n>We formalize a suite of tasks centered on user internal states, including affective experience, interactive intent, and cognitive memory.
arXiv Detail & Related papers (2026-05-17T05:05:29Z) - EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT [56.24624833924252]
EgoThinker is a framework that endows MLs with robust egocentric reasoning capabilities through-temporal chain-of-thought supervision and a two-stage learning curriculum.<n>EgoThinker outperforms existing methods across multiple egocentric benchmarks, while achieving substantial improvements in fine-grained-temporal localization tasks.
arXiv Detail & Related papers (2025-10-27T17:38:17Z) - EgoLife: Towards Egocentric Life Assistant [60.51196061794498]
We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses.<n>We conduct a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references.<n>This effort resulted in the EgoLife dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation.<n>We introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide
arXiv Detail & Related papers (2025-03-05T18:54:16Z) - EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering [95.2396264550978]
We introduce EgoTextVQA, a novel and rigorously constructed benchmark for egocentric QA assistance involving scene text.<n>EgoTextVQA contains 1.5K ego-view videos and 7K scene-text aware questions that reflect real user needs in outdoor driving and indoor house-keeping activities.
arXiv Detail & Related papers (2025-02-11T09:45:06Z) - Egocentric and Exocentric Methods: A Short Survey [25.41820386246096]
Egocentric vision captures the scene from the point of view of the camera wearer.<n>Exocentric vision captures the overall scene context.<n>Jointly modeling ego and exo views is crucial to developing next-generation AI agents.
arXiv Detail & Related papers (2024-10-27T22:38:51Z) - Retrieval-Augmented Egocentric Video Captioning [53.2951243928289]
EgoInstructor is a retrieval-augmented multimodal captioning model that automatically retrieves semantically relevant third-person instructional videos.
We train the cross-view retrieval module with a novel EgoExoNCE loss that pulls egocentric and exocentric video features closer by aligning them to shared text features that describe similar actions.
arXiv Detail & Related papers (2024-01-01T15:31:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.