EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding

• URL: http://arxiv.org/abs/2606.24422v1
• Date: Tue, 23 Jun 2026 10:59:25 GMT
• Title: EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding
• Abstract summary: EgoSAT is the first comprehensive benchmark for egocentric video reasoning in streaming settings.<n>It is designed to evaluate the capabilities of modern vision-language models (VLMs)<n>EgoSAT contains 1,997 unique videos spanning 165 hours of egocentric footage and around 4,800 high-quality question-answer pairs.
• Score: 12.811023931580523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce EgoSAT, the first comprehensive benchmark for egocentric video reasoning in streaming settings, designed to evaluate the capabilities of modern vision-language models (VLMs). The benchmark targets streaming interaction understanding, where video frames arrive sequentially and models must continuously interpret evolving visual context. EgoSAT unifies several previously distinct tasks within a single streaming framework. In this formulation, queries about completed events correspond to retrospective reasoning, queries about ongoing activities require online understanding, and queries about future actions involve prospective anticipation. This unified setting requires models to reason about the past, present, and future while operating under the constraint that only previously observed frames are available. EgoSAT contains 1,997 unique videos spanning 165 hours of egocentric footage and around 4,800 high-quality question-answer pairs, carefully designed to probe reasoning across varying temporal contexts. Using this benchmark, we evaluate a diverse set of both open-weight and closed-weight VLMs, providing a systematic assessment of their ability for streaming interaction understanding. By distinguishing answerability and conducting diagnostics on confidence of models, we find existing models not only struggle with prospective and retrospective modeling, but also exhibit severe mis-calibration: confidence often fails to track inherent answerability, leading to dangerous "confidently wrong" behaviors. Project page: https://leiyj23.github.io/EgoSAT/

Related papers

• VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026 [70.67609435035631]
  Given an egocentric video timestamp, the task requires anticipating the next human-object interaction.<n>ViSTA follows a StillFast-style design that combines object-centric spatial detection with short-horizon temporal context.<n>Experiments show that VISTA achieves first place in the EgoVis 2026 Ego4D STA Challenge.
  arXiv  Detail & Related papers  (2026-05-20T08:42:56Z)
• EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs [23.973683864983272]
  EgoT-Bench is a fine-grained egocentric benchmark for grounded and verifiable operation-centric reasoning.<n>EgoT-Bench comprises 3,172 QA pairs over 351 egocentric retrospect videos.
  arXiv  Detail & Related papers  (2026-05-19T09:02:20Z)
• How Far Are Video Models from True Multimodal Reasoning? [32.10821745418955]
  CLVG-Bench is an evaluation framework designed to probe video models' zero-shot reasoning capabilities.<n>It comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories.<n>We propose an Adaptive Video Evaluator (AVE) that aligns with human expert perception using minimal annotations.
  arXiv  Detail & Related papers  (2026-04-21T08:04:02Z)
• From Segments to Scenes: Temporal Understanding in Autonomous Driving via Vision-Language Model [13.666165551248701]
  Temporal understanding in autonomous driving (AD) remains a significant challenge.<n>Prior work has introduced datasets and benchmarks aimed at improving temporal reasoning.<n>No existing benchmark focuses exclusively on unique challenges of temporal understanding in ego-centric AD footage.
  arXiv  Detail & Related papers  (2025-12-04T21:57:10Z)
• StreamEQA: Towards Streaming Video Understanding for Embodied Scenarios [33.70462645363648]
  StreamEQA is the first benchmark for streaming video question answering in embodied scenarios.<n>It is built upon 156 independent long videos and generates approximately 21K question-answer pairs with precise timestamps.<n>We hope StreamEQA will catalyze research on streaming video understanding for embodied applications.
  arXiv  Detail & Related papers  (2025-12-04T04:48:16Z)
• Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models [56.851611990473174]
  Reasoning over dynamic visual content remains a central challenge for large language models.<n>We propose a reinforcement learning approach that enhances both temporal precision and reasoning consistency.<n>The resulting model, Video R2, achieves consistently higher TAC, VAS, and accuracy across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-11-28T18:59:58Z)
• ImplicitQA: Going beyond frames towards Implicit Video Reasoning [39.63171940350552]
  ImplicitQA is a novel benchmark designed to test VideoQA models on human-like implicit reasoning.<n>ImplicitQA comprises 1K meticulously annotated QA pairs drawn from 1K high-quality creative video clips.
  arXiv  Detail & Related papers  (2025-06-26T19:53:54Z)
• Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos [51.8995932557911]
  EgoTempo is a dataset designed to evaluate temporal understanding in the egocentric domain.<n>We show that state-of-the-art Multi-Modal Large Language Models (MLLMs) on benchmarks achieve remarkably high performance using just text or a single frame as input.<n>We hope EgoTempo will catalyze new research in the field and inspire models that better capture the complexity of temporal dynamics.
  arXiv  Detail & Related papers  (2025-03-17T18:50:36Z)
• TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models [75.42002690128486]
  TemporalBench is a new benchmark dedicated to evaluating fine-grained temporal understanding in videos. It consists of 10K video question-answer pairs, derived from 2K high-quality human annotations detailing the temporal dynamics in video clips. Results show that state-of-the-art models like GPT-4o achieve only 38.5% question answering accuracy on TemporalBench.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• Harnessing Temporal Causality for Advanced Temporal Action Detection [53.654457142657236]
  We introduce CausalTAD, which combines causal attention and causal Mamba to achieve state-of-the-art performance on benchmarks. We ranked 1st in the Action Recognition, Action Detection, and Audio-Based Interaction Detection tracks at the EPIC-Kitchens Challenge 2024, and 1st in the Moment Queries track at the Ego4D Challenge 2024.
  arXiv  Detail & Related papers  (2024-07-25T06:03:02Z)
• Self-Regulated Learning for Egocentric Video Activity Anticipation [147.9783215348252]
  Self-Regulated Learning (SRL) aims to regulate the intermediate representation consecutively to produce representation that emphasizes the novel information in the frame of the current time-stamp. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets.
  arXiv  Detail & Related papers  (2021-11-23T03:29:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.