EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning

• URL: http://arxiv.org/abs/2603.09731v2
• Date: Thu, 12 Mar 2026 12:40:05 GMT
• Title: EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning
• Abstract summary: We study whether multimodal large language models can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint.<n>We introduce EXPLORE-Bench, a benchmark curated from real first-person videos spanning diverse scenarios.<n>Experiments on a range of proprietary and open-source MLLMs reveal a significant performance gap to humans.
• Score: 63.010793398283134
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) are increasingly considered as a foundation for embodied agents, yet it remains unclear whether they can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint. We study this gap through a new task, Egocentric Scene Prediction with LOng-horizon REasoning: given an initial-scene image and a sequence of atomic action descriptions, a model is asked to predict the final scene after all actions are executed. To enable systematic evaluation, we introduce EXPLORE-Bench, a benchmark curated from real first-person videos spanning diverse scenarios. Each instance pairs long action sequences with structured final-scene annotations, including object categories, visual attributes, and inter-object relations, which supports fine-grained, quantitative assessment. Experiments on a range of proprietary and open-source MLLMs reveal a significant performance gap to humans, indicating that long-horizon egocentric reasoning remains a major challenge. We further analyze test-time scaling via stepwise reasoning and show that decomposing long action sequences can improve performance to some extent, while incurring non-trivial computational overhead. Overall, EXPLORE-Bench provides a principled testbed for measuring and advancing long-horizon reasoning for egocentric embodied perception.

Related papers

• ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes? [43.007245223642606]
  ViSTR-Bench is an evaluation suite designed to systematically assess whether MLLMs can perform qualitative reasoning from continuous visual cues in dynamic scenes.<n>The benchmark comprises 15 distinct subtasks and 1,340 high-quality video question-answer pairs spanning diverse tabletop, indoor, and outdoor scenarios.
  arXiv  Detail & Related papers  (2026-07-23T02:49:24Z)
• EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs [23.973683864983272]
  EgoT-Bench is a fine-grained egocentric benchmark for grounded and verifiable operation-centric reasoning.<n>EgoT-Bench comprises 3,172 QA pairs over 351 egocentric retrospect videos.
  arXiv  Detail & Related papers  (2026-05-19T09:02:20Z)
• Self-Consistent Latent Reasoning: Long Latent Sequence Reasoning for Vision-Language Model [56.21523258053447]
  SCOLAR (Self-COnsistent LAtent Reasoning) introduces a lightweight detransformer that generates auxiliary visual tokens in a single shot.<n>SCOLAR extends acceptable latent CoT length by over $30times$, achieves state-of-the-art among open-source models on real-world reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-05-12T14:13:08Z)
• Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models [154.39583176906893]
  Large language models (MLs) have advanced understanding, they remain highly prone to hallucinations in scenes.<n>We argue this stems from a failure intemporal monitoring, the ability to persistently track object dynamic identities, states, and relations over time.<n>Existing benchmarks obscure this deficit relying on single final-answer queries that can often be resolved via local visual cues or statistical priors.
  arXiv  Detail & Related papers  (2026-05-09T14:32:36Z)
• From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning [79.65573577667944]
  We present EgoTSR, a curriculum-based framework for learning task-basedtemporal reasoning.<n>EgoTSR is built on the premise embodied reasoning should evolve from explicit spatial understanding to task-state assessment.<n>Experiments demonstrate that EgoTSR effectively eliminates biases, achieving 92.4% accuracy on long-horizon logical reasoning tasks.
  arXiv  Detail & Related papers  (2026-04-12T08:14:49Z)
• Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention [58.05340906967343]
  Egocentric Referring Video Object (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos.<n>Existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets.<n>We introduce Causal-REferring (CERES), a plug-in causal framework that adapts strong, pre-trained RVOSs to the egocentric domain.
  arXiv  Detail & Related papers  (2025-12-30T16:22:14Z)
• Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video? [1.1288535170985818]
  We introduce AAG, a method for Action Anticipation at a Glimpse.<n>AAG combines RGB features with depth cues from a single frame for enhanced spatial reasoning.<n>Our results demonstrate that multimodal single-frame action anticipation using AAG can perform competitively.
  arXiv  Detail & Related papers  (2025-12-02T14:57:17Z)
• HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding [120.84817886550765]
  Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks involving both images and videos.<n>Existing human-centric benchmarks predominantly emphasize video generation quality and action recognition, while overlooking essential perceptual and cognitive abilities required in human-centered scenarios.<n>We propose a rigorously curated benchmark designed to provide a more holistic evaluation of MLLMs in human-centric video understanding.
  arXiv  Detail & Related papers  (2025-07-07T11:52:24Z)
• STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training [87.58996020705258]
  Video Large Language Models (Video-LLMs) have recently shown strong derivation in basic video understanding tasks.<n>Video-LLMs struggle with compositional reasoning that requires multi-step explicit-temporal inference across object relations, interactions and events.<n>We propose STEP, a novel graph-guided self-training method that enables VideoLLMs to generate reasoning-rich finetuning data from any raw videos to improve itself.
  arXiv  Detail & Related papers  (2024-11-29T11:54:55Z)
• PALM: Predicting Actions through Language Models [74.10147822693791]
  We introduce PALM, an approach that tackles the task of long-term action anticipation. Our method incorporates an action recognition model to track previous action sequences and a vision-language model to articulate relevant environmental details. Our experimental results demonstrate that PALM surpasses the state-of-the-art methods in the task of long-term action anticipation.
  arXiv  Detail & Related papers  (2023-11-29T02:17:27Z)
• Object-centric Video Representation for Long-term Action Anticipation [33.115854386196126]
  Key motivation is that objects provide important cues to recognize and predict human-object interactions. We propose to build object-centric video representations by leveraging visual-language pretrained models. To recognize and predict human-object interactions, we use a Transformer-based neural architecture.
  arXiv  Detail & Related papers  (2023-10-31T22:54:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.