Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphs

• URL: http://arxiv.org/abs/2606.25842v2
• Date: Thu, 02 Jul 2026 12:05:36 GMT
• Title: Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphs
• Abstract summary: We introduce a framework for egocentric video question answering (VQA) that overcomes these input constraints through Egocentric Scene Graphs (EgoSGs)<n>By representing videos as compact, text-based scene graphs, our method preserves the essential visual and temporal information of the original video in a symbolic form.<n>On HD-EPIC VQA, our method achieves state-of-the-art results, outperforming strong video-based baselines on multiple models.
• Score: 12.707862116540914
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing multi-modal large language models (MLLMs) face significant challenges in processing long video sequences due to strict input token limitations. As a result, current video understanding approaches, especially in egocentric settings characterized by complex dynamics, frequent state changes, and moving cameras, are forced to massively subsample frames. This leads to severe loss of temporal and contextual information, constraining their ability to perform fine-grained video reasoning. In this work, we introduce a framework for egocentric video question answering (VQA) that overcomes these input constraints through Egocentric Scene Graphs (EgoSGs), i.e., temporally grounded, structured representations that capture objects, attributes, spatial relations, and interactions over time. By representing videos as compact, text-based scene graphs, our method preserves the essential visual and temporal information of the original video in a symbolic form that drastically reduces input length while maintaining semantic richness. Crucially, this enables MLLMs to reason efficiently over entire video sequences within their token budget. On HD-EPIC VQA, our method achieves state-of-the-art results, outperforming strong video-based baselines on multiple models and suggesting that structured, temporally grounded representations like EgoSGs can bridge long-form egocentric video understanding and the context limitations of today's MLLMs.

Related papers

• EgoGraph: Temporal Knowledge Graph for Egocentric Video Understanding [11.51428438970598]
  EgoGraph is a training-free and dynamic knowledge-graph construction framework that explicitly encodes long-term, cross-entity dependencies in egocentric video streams.<n>We develop a temporal relational modeling strategy that captures temporal dependencies across entities and accumulates stable long-term memory over multiple days, enabling complex temporal reasoning.
  arXiv  Detail & Related papers  (2026-02-27T06:20:58Z)
• Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding [56.45689495743107]
  Vgent is a graph-based retrieval-reasoning-augmented generation framework to enhance LVLMs for long video understanding.<n>We evaluate our framework with various open-source LVLMs on three long-video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-10-15T19:14:58Z)
• Time-Scaling State-Space Models for Dense Video Captioning [29.405515743544687]
  State-Space Models with Transfer State is a time-scaling model for dense video captioning.<n>It is suitable for generating captions on-the-fly, in an online or streaming manner, without having to wait for the full video to be processed.<n>When applied to dense video captioning, our approach scales well with video lengths and uses 7x fewer FLOPs.
  arXiv  Detail & Related papers  (2025-09-03T15:56:20Z)
• When and What: Diffusion-Grounded VideoLLM with Entity Aware Segmentation for Long Video Understanding [12.410012029024342]
  We present Grounded VideoDiT, a Video LLM designed to overcome limitations by introducing three key innovations.<n>First, a Diffusion Temporal Latent (DTL) encoder enhances boundary sensitivity and maintains temporal consistency.<n>Second, object grounded representations explicitly bind query entities to localized visual evidence, strengthening alignment.<n>Third, a mixed token scheme with discrete temporal timestamp tokens provides explicit modeling, enabling fine grained temporal reasoning.
  arXiv  Detail & Related papers  (2025-08-21T15:12:14Z)
• Episodic Memory Representation for Long-form Video Understanding [52.33907540905242]
  Large Video Language Models excel at general video understanding but struggle with long-form context window limits.<n>We introduce Video-EM, a training free framework inspired by the principles of human memory.<n>Video-EM achieves performance gains of 4-9 percent over respective baselines while utilizing fewer frames.
  arXiv  Detail & Related papers  (2025-08-13T04:33:07Z)
• RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graph [3.1671311914949545]
  RAVU is a framework for video enhanced understanding by retrieval with reasoning over atemporal graph.<n>We construct a graph representation of capturing the video both spatial and temporal relationships between entities.<n>To answer complex queries, we decompose the queries into a sequence of reasoning steps and execute these steps on the graph.<n>Our approach enables more accurate understanding of long videos, particularly for queries that require multi-hop reasoning and tracking objects across frames.
  arXiv  Detail & Related papers  (2025-05-06T04:38:09Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding [65.46303012350207]
  LongVU is an adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. We leverage DINOv2 features to remove redundant frames that exhibit high similarity. We perform spatial token reduction across frames based on their temporal dependencies.
  arXiv  Detail & Related papers  (2024-10-22T21:21:37Z)
• Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models [53.235170710385006]
  We introduce Grounded-VideoLLM, a novel Video-LLM adept at perceiving and reasoning over specific video moments in a fine-grained manner.<n>We sharpen our model by incorporating (1) an additional temporal stream to encode the relationships between frames and (2) discrete temporal tokens enriched with specific time knowledge.<n>In experiments, Grounded-VideoLLM excels in fine-grained grounding tasks such as temporal sentence grounding, dense video captioning, and grounded VideoQA.
  arXiv  Detail & Related papers  (2024-10-04T10:04:37Z)
• Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization [52.63845811751936]
  Video pre-training is challenging due to the modeling of its dynamics video. In this paper, we address such limitations in video pre-training with an efficient video decomposition. Our framework is both capable of comprehending and generating image and video content, as demonstrated by its performance across 13 multimodal benchmarks.
  arXiv  Detail & Related papers  (2024-02-05T16:30:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.