EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning

• URL: http://arxiv.org/abs/2605.17262v1
• Date: Sun, 17 May 2026 05:05:29 GMT
• Title: EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning
• Abstract summary: EgoIntrospect is the first egocentric dataset captured in user-driven scenarios with self-annotations.<n>It consists of 180 hours of recordings from 60 subjects, with an average recording duration of 3 hours per subject.<n>We formalize a suite of tasks centered on user internal states, including affective experience, interactive intent, and cognitive memory.
• Score: 47.853306116245484
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Despite extensive efforts on egocentric video datasets and benchmarks, understanding users' internal states, which is crucial for enabling seamless AI assistant experiences, remains largely overlooked. In this work, we introduce EgoIntrospect, the first egocentric dataset captured in user-driven scenarios with self-annotations that explicitly reveal users' interactive intentions with AI assistants. EgoIntrospect was collected using a cross-device setup, providing synchronized video, audio, gaze, motion, and physiological signals. It consists of 180 hours of recordings from 60 subjects, with an average recording duration of 3 hours per subject. Leveraging EgoIntrospect, we formalize a suite of tasks centered on user internal states, including affective experience, interactive intent, and cognitive memory. We further process the annotations to construct benchmarks that evaluate the ability of modern multimodal large language models to reason about users' internal states from egocentric observations. Experiments on our benchmark suggest that existing multimodal large language models struggle to effectively leverage multimodal signals to infer users' subjective internal states. The dataset and annotations will be made publicly available to advance research in egocentric vision and wearable AI assistants. Project page: https://ego-introspect.github.io/

Related papers

• EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports [5.919102792488079]
  We introduce EgoArgus, a human-annotated dataset for evaluating egocentric assistants on understanding and decision tasks.<n>Our results demonstrate that it is still challenging for current VLMs as reliable egocentric assistants.
  arXiv  Detail & Related papers  (2026-08-26T09:09:30Z)
• EgoSelf: From Memory to Personalized Egocentric Assistant [44.86486625757587]
  EgoSelf is a system that includes a graph-based interaction memory constructed from past observations.<n>The memory captures temporal and semantic relationships among interaction events and entities, from which user-specific profiles are derived.<n>The personalized learning task is formulated as a prediction problem where the model predicts possible future interactions from individual user's historical behavior recorded in the graph.
  arXiv  Detail & Related papers  (2026-04-21T15:15:02Z)
• Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention [58.05340906967343]
  Egocentric Referring Video Object (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos.<n>Existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets.<n>We introduce Causal-REferring (CERES), a plug-in causal framework that adapts strong, pre-trained RVOSs to the egocentric domain.
  arXiv  Detail & Related papers  (2025-12-30T16:22:14Z)
• Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance [19.081509891187114]
  We present Ego-EXTRA, a video-language Egocentric dataset for EXpert-TRAinee assistance.<n>Ego-EXTRA features 50 hours of unscripted egocentric videos of subjects performing procedural activities.<n>Real-world experts provide guidance and answer specific questions using natural language.
  arXiv  Detail & Related papers  (2025-12-15T11:53:35Z)
• TeleEgo: Benchmarking Egocentric AI Assistants in the Wild [55.53194302888826]
  Egocentric AI assistants in real-world settings must process multi-modal inputs (video, audio, text)<n>We introduce textbfTeleEgo, a long-duration, streaming, omni-modal benchmark for evaluating egocentric AI assistants.<n>The dataset features over 14 hours per participant of synchronized egocentric video, audio, and text across four domains.
  arXiv  Detail & Related papers  (2025-10-28T01:24:24Z)
• EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT [56.24624833924252]
  EgoThinker is a framework that endows MLs with robust egocentric reasoning capabilities through-temporal chain-of-thought supervision and a two-stage learning curriculum.<n>EgoThinker outperforms existing methods across multiple egocentric benchmarks, while achieving substantial improvements in fine-grained-temporal localization tasks.
  arXiv  Detail & Related papers  (2025-10-27T17:38:17Z)
• Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions [110.43343503158306]
  This paper embeds the manual-assisted task into a vision-language-action framework, where the assistant provides services to the instructor following egocentric vision and commands.<n>Under this setting, we accomplish InterVLA, the first large-scale human-object-human interaction dataset with 11.4 hours and 1.2M frames of multimodal data.<n>We establish novel benchmarks on egocentric human motion estimation, interaction synthesis, and interaction prediction with comprehensive analysis.
  arXiv  Detail & Related papers  (2025-08-06T17:46:23Z)
• EgoLife: Towards Egocentric Life Assistant [60.51196061794498]
  We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses.<n>We conduct a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references.<n>This effort resulted in the EgoLife dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation.<n>We introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide
  arXiv  Detail & Related papers  (2025-03-05T18:54:16Z)
• Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions? [48.702973928321946]
  Egocentric video-language pretraining is a crucial step in advancing the understanding of hand-object interactions in first-person scenarios.<n>Despite successes on existing testbeds, we find that current EgoVLMs can be easily misled by simple modifications.<n>This raises the question: Do EgoVLMs truly understand hand-object interactions?
  arXiv  Detail & Related papers  (2024-05-28T00:27:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.