EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environment

• URL: http://arxiv.org/abs/2604.08342v1
• Date: Thu, 09 Apr 2026 15:13:36 GMT
• Title: EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environment
• Abstract summary: EgoEverything is a benchmark that explicitly considers human behavior by leveraging human attention signals, abstracted from gaze data, when generating questions.<n>It comprises over 5,000 multiple choice question answer pairs, spanning more than 100 hours of video.<n>It more faithfully captures natural human behavior and offers a realistic evaluation setting for long-context egocentric video understanding in AR.
• Score: 13.283186212993703
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Long context egocentric video understanding has recently attracted significant research attention, with augmented reality (AR) highlighted as one of its most important application domains. Nevertheless, the task remains highly challenging due to the need for reasoning over extended temporal contexts and diverse, unstructured activities. Although several benchmarks exist, most egocentric datasets rely on human worn cameras and focus mainly on visual content, with limited consideration of underlying user behavior when forming video-related queries. EgoEverything is a benchmark that explicitly considers human behavior by leveraging human attention signals, abstracted from gaze data, when generating questions. It comprises over 5,000 multiple choice question answer pairs, spanning more than 100 hours of video. By integrating human attention signals during question generation, it more faithfully captures natural human behavior and offers a realistic evaluation setting for long-context egocentric video understanding in AR.

Related papers

• LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension [41.229006294351315]
  LongEgoRefer is a novel benchmark constructed from long-form videos in the Ego4D dataset.<n>It exhibits detailed linguistic descriptions, and complex human-object interactions embedded in long, dynamic egocentric narratives.<n>It defines a demanding-temporal grounding problem that requires models to identify both intrinsic when an event occurs and where the referred object appears within extended video sequences.
  arXiv  Detail & Related papers  (2026-07-02T12:32:53Z)
• EgoGraph: Temporal Knowledge Graph for Egocentric Video Understanding [11.51428438970598]
  EgoGraph is a training-free and dynamic knowledge-graph construction framework that explicitly encodes long-term, cross-entity dependencies in egocentric video streams.<n>We develop a temporal relational modeling strategy that captures temporal dependencies across entities and accumulates stable long-term memory over multiple days, enabling complex temporal reasoning.
  arXiv  Detail & Related papers  (2026-02-27T06:20:58Z)
• Learning Situated Awareness in the Real World [63.75211123289058]
  SAW-Bench is a novel benchmark for evaluating egocentric situated awareness using real-world videos.<n>It probes a model's observer-centric understanding with six different awareness tasks.<n>Our comprehensive evaluation reveals a human-model performance gap of 37.66%, even with the best-performing MFM, Gemini 3 Flash.
  arXiv  Detail & Related papers  (2026-02-18T18:22:52Z)
• Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention [58.05340906967343]
  Egocentric Referring Video Object (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos.<n>Existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets.<n>We introduce Causal-REferring (CERES), a plug-in causal framework that adapts strong, pre-trained RVOSs to the egocentric domain.
  arXiv  Detail & Related papers  (2025-12-30T16:22:14Z)
• EgoM2P: Egocentric Multimodal Multitask Pretraining [55.259234688003545]
  Building large-scale egocentric multimodal and multitask models presents unique challenges.<n> EgoM2P is a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding.<n>We will fully open-source EgoM2P to support the community and advance egocentric vision research.
  arXiv  Detail & Related papers  (2025-06-09T15:59:25Z)
• HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos [2.6749843984691672]
  We present HiERO, a weakly-supervised method to enrich video segments features with hierarchical activity threads.<n>By aligning video clips with their narrated descriptions, HiERO infers contextual, semantic and temporal reasoning with an hierarchical architecture.
  arXiv  Detail & Related papers  (2025-05-19T09:47:41Z)
• EgoLife: Towards Egocentric Life Assistant [60.51196061794498]
  We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses.<n>We conduct a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references.<n>This effort resulted in the EgoLife dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation.<n>We introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide
  arXiv  Detail & Related papers  (2025-03-05T18:54:16Z)
• EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World [12.699670048897085]
  In human imitation learning, the imitator typically take the egocentric view as a benchmark, naturally transferring behaviors observed from an exocentric view to their owns.<n>We introduce EgoMe, which towards following the process of human imitation learning via the imitator's egocentric view in the real world.<n>Our dataset includes 7902 paired exo-ego videos spanning diverse daily behaviors in various real-world scenarios.
  arXiv  Detail & Related papers  (2025-01-31T11:48:22Z)
• Egocentric and Exocentric Methods: A Short Survey [25.41820386246096]
  Egocentric vision captures the scene from the point of view of the camera wearer.<n>Exocentric vision captures the overall scene context.<n>Jointly modeling ego and exo views is crucial to developing next-generation AI agents.
  arXiv  Detail & Related papers  (2024-10-27T22:38:51Z)
• AMEGO: Active Memory from long EGOcentric videos [26.04157621755452]
  We introduce AMEGO, a novel approach aimed at enhancing the comprehension of very-long egocentric videos. Inspired by the human's ability to maintain information from a single watching, AMEGO focuses on constructing a self-contained representations from one egocentric video. This representation is semantic-free and facilitates multiple queries without the need to reprocess the entire visual content.
  arXiv  Detail & Related papers  (2024-09-17T06:18:47Z)
• EgoTaskQA: Understanding Human Tasks in Egocentric Videos [89.9573084127155]
  EgoTaskQA benchmark provides home for crucial dimensions of task understanding through question-answering on real-world egocentric videos. We meticulously design questions that target the understanding of (1) action dependencies and effects, (2) intents and goals, and (3) agents' beliefs about others. We evaluate state-of-the-art video reasoning models on our benchmark and show their significant gaps between humans in understanding complex goal-oriented egocentric videos.
  arXiv  Detail & Related papers  (2022-10-08T05:49:05Z)
• Exploring Anchor-based Detection for Ego4D Natural Language Query [74.87656676444163]
  This paper presents technique report of Ego4D natural language query challenge in CVPR 2022. We propose our solution of this challenge to solve the above issues.
  arXiv  Detail & Related papers  (2022-08-10T14:43:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.