EgoIntent: An Egocentric Step-level Benchmark for Understanding What, Why, and Next

• URL: http://arxiv.org/abs/2603.12147v1
• Date: Thu, 12 Mar 2026 16:46:01 GMT
• Title: EgoIntent: An Egocentric Step-level Benchmark for Understanding What, Why, and Next
• Abstract summary: EgoIntent is a step-level intent understanding benchmark for egocentric videos.<n>It comprises 3,014 steps spanning 15 diverse indoor and outdoor daily-life scenarios.<n>Each clip is truncated immediately before the key outcome of the queried step.
• Score: 52.87513180819888
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable video reasoning capabilities across diverse tasks. However, their ability to understand human intent at a fine-grained level in egocentric videos remains largely unexplored. Existing benchmarks focus primarily on episode-level intent reasoning, overlooking the finer granularity of step-level intent understanding. Yet applications such as intelligent assistants, robotic imitation learning, and augmented reality guidance require understanding not only what a person is doing at each step, but also why and what comes next, in order to provide timely and context-aware support. To this end, we introduce EgoIntent, a step-level intent understanding benchmark for egocentric videos. It comprises 3,014 steps spanning 15 diverse indoor and outdoor daily-life scenarios, and evaluates models on three complementary dimensions: local intent (What), global intent (Why), and next-step plan (Next). Crucially, each clip is truncated immediately before the key outcome of the queried step (e.g., contact or grasp) occurs and contains no frames from subsequent steps, preventing future-frame leakage and enabling a clean evaluation of anticipatory step understanding and next-step planning. We evaluate 15 MLLMs, including both state-of-the-art closed-source and open-source models. Even the best-performing model achieves an average score of only 33.31 across the three intent dimensions, underscoring that step-level intent understanding in egocentric videos remains a highly challenging problem that calls for further investigation.

Related papers

• Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering [81.15678825882692]
  EgoPointVQA is a dataset and benchmark for gesture-grounded egocentric question answering.<n>Hand Intent Tokens (HINT) encodes tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model.<n>We show that our model outperforms others in different backbones and model sizes.
  arXiv  Detail & Related papers  (2026-03-13T00:23:38Z)
• EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning [63.010793398283134]
  We study whether multimodal large language models can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint.<n>We introduce EXPLORE-Bench, a benchmark curated from real first-person videos spanning diverse scenarios.<n>Experiments on a range of proprietary and open-source MLLMs reveal a significant performance gap to humans.
  arXiv  Detail & Related papers  (2026-03-10T14:33:44Z)
• HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos [2.6749843984691672]
  We present HiERO, a weakly-supervised method to enrich video segments features with hierarchical activity threads.<n>By aligning video clips with their narrated descriptions, HiERO infers contextual, semantic and temporal reasoning with an hierarchical architecture.
  arXiv  Detail & Related papers  (2025-05-19T09:47:41Z)
• EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos [26.930652137352197]
  We introduce EgoToM, a new video question-answering benchmark that extends Theory-of-Mind evaluation to egocentric domains.<n>Using a causal ToM model, we generate multi-choice video QA instances for the Ego4D dataset to benchmark the ability to predict a camera wearer's goals, beliefs, and next actions.<n>We study the performance of both humans and state of the art multimodal large language models (MLLMs) on these three interconnected inference problems.
  arXiv  Detail & Related papers  (2025-03-28T05:10:59Z)
• Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding [69.96199605596138]
  Current MLLMs primarily focus on third-person (exocentric) vision, overlooking the unique aspects of first-person (egocentric) videos.<n>We propose learning the mapping between exocentric and egocentric domains to enhance egocentric video understanding.<n>We introduce Ego-ExoClip, a pre-training dataset comprising 1.1M synchronized ego-exo clip-text pairs.
  arXiv  Detail & Related papers  (2025-03-12T08:10:33Z)
• MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA [72.47344411599322]
  This research aims to explore building a multimodal foundation model for egocentric video understanding.<n>We automatically generate 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long in Ego4D based on human-annotated data.<n>We contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths.
  arXiv  Detail & Related papers  (2024-10-09T17:59:59Z)
• EgoTaskQA: Understanding Human Tasks in Egocentric Videos [89.9573084127155]
  EgoTaskQA benchmark provides home for crucial dimensions of task understanding through question-answering on real-world egocentric videos. We meticulously design questions that target the understanding of (1) action dependencies and effects, (2) intents and goals, and (3) agents' beliefs about others. We evaluate state-of-the-art video reasoning models on our benchmark and show their significant gaps between humans in understanding complex goal-oriented egocentric videos.
  arXiv  Detail & Related papers  (2022-10-08T05:49:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.