Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding

• URL: http://arxiv.org/abs/2605.15342v1
• Date: Thu, 14 May 2026 19:12:20 GMT
• Title: Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding
• Abstract summary: Minerva-Ego is a benchmark for evaluating egocentric visual reasoning.<n> Benchmarking experiments show that state-of-the-art models still have a large gap to human performance.
• Score: 54.691865975180576
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of intermediate reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a benchmark for evaluating complex egocentric visual reasoning. We extend recent high-quality video data sources recorded from egocentric / embodied settings with a set of challenging, multi-step multimodal questions and spatiotemporally-dense human-annotated reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each reasoning trace in the dataset with the objects of interest required to solve the question, as spatiotemporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and 'when' to look yields substantial improvements in performance. Minerva-Ego can be downloaded at https://github.com/google-deepmind/neptune.

Related papers

• EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs [23.973683864983272]
  EgoT-Bench is a fine-grained egocentric benchmark for grounded and verifiable operation-centric reasoning.<n>EgoT-Bench comprises 3,172 QA pairs over 351 egocentric retrospect videos.
  arXiv  Detail & Related papers  (2026-05-19T09:02:20Z)
• EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning [47.853306116245484]
  EgoIntrospect is the first egocentric dataset captured in user-driven scenarios with self-annotations.<n>It consists of 180 hours of recordings from 60 subjects, with an average recording duration of 3 hours per subject.<n>We formalize a suite of tasks centered on user internal states, including affective experience, interactive intent, and cognitive memory.
  arXiv  Detail & Related papers  (2026-05-17T05:05:29Z)
• Do MLLMs Understand Pointing? Benchmarking and Enhancing Referential Reasoning in Egocentric Vision [22.502853661316028]
  EgoPoint-Bench is a comprehensive question-answering benchmark designed to evaluate and enhance multimodal pointing reasoning in egocentric views.<n>We show that models fine-tuned on our synthetic data achieve significant performance gains and robust sim-to-real generalization.<n>This work highlights the importance of spatially aware supervision and offers a scalable path toward precise egocentric AI assistants.
  arXiv  Detail & Related papers  (2026-04-23T09:15:42Z)
• EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning [63.010793398283134]
  We study whether multimodal large language models can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint.<n>We introduce EXPLORE-Bench, a benchmark curated from real first-person videos spanning diverse scenarios.<n>Experiments on a range of proprietary and open-source MLLMs reveal a significant performance gap to humans.
  arXiv  Detail & Related papers  (2026-03-10T14:33:44Z)
• EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT [56.24624833924252]
  EgoThinker is a framework that endows MLs with robust egocentric reasoning capabilities through-temporal chain-of-thought supervision and a two-stage learning curriculum.<n>EgoThinker outperforms existing methods across multiple egocentric benchmarks, while achieving substantial improvements in fine-grained-temporal localization tasks.
  arXiv  Detail & Related papers  (2025-10-27T17:38:17Z)
• Fine-grained Spatiotemporal Grounding on Egocentric Videos [13.319346673043286]
  We introduce EgoMask, the first pixel-level benchmark for fine-temporal grounding in egocentric videos.<n>EgoMask is constructed by our proposed automatic annotation pipeline, which annotates referring expressions and object masks.<n>We also create EgoMask-Train, a large-scale training dataset to facilitate model development.
  arXiv  Detail & Related papers  (2025-08-01T10:53:27Z)
• MINERVA: Evaluating Complex Video Reasoning [72.12644008002566]
  We provide a new video reasoning dataset called MINERVA for modern multimodal models.<n>Our dataset is multimodal, diverse in terms of video domain and length, and consists of complex multi-step questions.<n>We perform fine-grained error analysis to identify common failure modes across various models, and create a taxonomy of reasoning errors.
  arXiv  Detail & Related papers  (2025-05-01T17:41:49Z)
• Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos [51.8995932557911]
  EgoTempo is a dataset designed to evaluate temporal understanding in the egocentric domain.<n>We show that state-of-the-art Multi-Modal Large Language Models (MLLMs) on benchmarks achieve remarkably high performance using just text or a single frame as input.<n>We hope EgoTempo will catalyze new research in the field and inspire models that better capture the complexity of temporal dynamics.
  arXiv  Detail & Related papers  (2025-03-17T18:50:36Z)
• MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA [72.47344411599322]
  This research aims to explore building a multimodal foundation model for egocentric video understanding.<n>We automatically generate 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long in Ego4D based on human-annotated data.<n>We contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths.
  arXiv  Detail & Related papers  (2024-10-09T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.