G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding

• URL: http://arxiv.org/abs/2608.20157v1
• Date: Thu, 20 Aug 2026 15:15:08 GMT
• Title: G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding
• Abstract summary: G3Ego is a graph-based framework for egocentric action understanding.<n>It prunes irrelevant entities using the camera wearer's gaze.<n>It produces efficient and interpretable representations focused on action-relevant interactions.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Egocentric action understanding is often addressed using large video models pretrained on extensive exocentric datasets. However, many first-person actions depend on a small number of hand-object interactions involving only a few relevant entities. We propose G3Ego, a graph-based framework for egocentric action understanding that uses gaze as a structural cue to identify action-relevant entities in the scene. From sparsely sampled frames, G3Ego constructs action scene graphs from vision-language descriptions, grounded objects, and hand cues, and then prunes irrelevant entities using the camera wearer's gaze. The resulting graph embeddings are temporally aggregated for action recognition and anticipation. Unlike prior work that uses gaze primarily as an auxiliary modality or attention signal, G3Ego incorporates gaze directly into graph construction, producing efficient and interpretable representations focused on action-relevant interactions. Experiments on EGTEA Gaze+ and MECCANO show that G3Ego achieves competitive performance compared with video-based approaches and consistently improves Macro-F1 under class-imbalanced evaluation, while avoiding reliance on computationally expensive video pretraining. These results demonstrate the effectiveness of gaze-guided graph representations for egocentric action understanding.

Related papers

• Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs [10.367002095946843]
  We argue that explicit, compositional and editable interactions can play a crucial role for rich activity understanding.<n>We introduce EgoEgo, a large scale annotation set extending Ego4D with annotated scene graphs.<n>We also formulate the activity-driven graph-edit forecasting (A-GEF) problem, a novel task that casts scene dynamics as a sequence of structured transformations.
  arXiv  Detail & Related papers  (2026-07-02T16:53:42Z)
• From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models [67.00247936789127]
  Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions.<n>We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs.
  arXiv  Detail & Related papers  (2026-06-13T18:00:42Z)
• OR-Action: Multi-Role Video Understanding with Fine-Grained Actions [37.96856087875045]
  Fine-grained understanding of operating room (OR) activity could enable workflow-aware assistance.<n>Current scene graph prediction methods struggle to model temporal structure.<n>We introduce a vision-only temporal model that outperforms graph-based methods.
  arXiv  Detail & Related papers  (2026-06-11T13:24:40Z)
• MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism [70.69809410471993]
  Current Vision-Language Models struggle with hours-long videos because processing full-length visual sequences induces prohibitive token explosion and attention dilution.<n>We introduce MemDreamer to decouple perception and reasoning, shifting long-video understanding into an agentic exploration process.<n>MemDreamer achieves SOTA results across four mainstream benchmarks, narrowing the gap with human experts to only 3.7 points.
  arXiv  Detail & Related papers  (2026-06-05T17:59:21Z)
• ArtiSG: Functional 3D Scene Graph Construction via Human-demonstrated Articulated Objects Manipulation [51.54082859171464]
  ArtiSG is a framework that constructs functional 3D scene graphs by encoding human demonstrations into structured robotic memory.<n>We show that ArtiSG significantly outperforms baselines in functional element recall and articulation estimation precision.
  arXiv  Detail & Related papers  (2025-12-31T13:10:40Z)
• Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention [58.05340906967343]
  Egocentric Referring Video Object (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos.<n>Existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets.<n>We introduce Causal-REferring (CERES), a plug-in causal framework that adapts strong, pre-trained RVOSs to the egocentric domain.
  arXiv  Detail & Related papers  (2025-12-30T16:22:14Z)
• OOTSM: A Decoupled Linguistic Framework for Effective Scene Graph Anticipation [14.938566273427098]
  Scene Graph Anticipation (SGA) involves predicting future scene graphs from video clips.<n>Existing SGA approaches leverage visual cues, often struggling to integrate valuable commonsense knowledge.<n>We propose a new approach to better understand the objects, concepts, and relationships in a scene graph.
  arXiv  Detail & Related papers  (2025-09-06T09:35:15Z)
• Action Scene Graphs for Long-Form Understanding of Egocentric Videos [23.058999979457546]
  We present Egocentric Action Scene Graphs (EASGs), a new representation for long-form understanding of egocentric videos. EASGs provide a temporally evolving graph-based description of the actions performed by the camera wearer. We will release the dataset and the code to replicate experiments and annotations.
  arXiv  Detail & Related papers  (2023-12-06T10:01:43Z)
• Iterative Graph Self-Distillation [161.04351580382078]
  We propose a novel unsupervised graph learning paradigm called Iterative Graph Self-Distillation (IGSD) IGSD iteratively performs the teacher-student distillation with graph augmentations. We show that we achieve significant and consistent performance gain on various graph datasets in both unsupervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-10-23T18:37:06Z)
• Egocentric Object Manipulation Graphs [8.759425622561334]
  Ego-OMG is a novel representation for activity and modeling anticipation of near future actions. It integrates semantic temporal structure, short-term dynamics, and representations for appearance. Code will be released upon acceptance of Ego-OMG.
  arXiv  Detail & Related papers  (2020-06-05T02:03:25Z)
• GoGNN: Graph of Graphs Neural Network for Predicting Structured Entity Interactions [70.9481395807354]
  We propose a Graph of Graphs Neural Network (GoGNN), which extracts the features in both structured entity graphs and the entity interaction graph in a hierarchical way. GoGNN outperforms the state-of-the-art methods on two representative structured entity interaction prediction tasks.
  arXiv  Detail & Related papers  (2020-05-12T03:46:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.