Keystep Recognition using Graph Neural Networks

• URL: http://arxiv.org/abs/2506.01102v1
• Date: Sun, 01 Jun 2025 17:54:58 GMT
• Title: Keystep Recognition using Graph Neural Networks
• Authors: Julia Lee Romero, Kyle Min, Subarna Tripathi, Morteza Karimzadeh,
• Abstract summary: We propose a flexible graph-learning framework for keystep recognition in egocentric videos.<n>The constructed graphs are sparse and computationally efficient, outperforming existing larger models substantially.<n>We perform extensive experiments on the Ego-Exo4D dataset and show that our proposed flexible graph-based framework notably outperforms existing methods.
• Score: 11.421362760480527
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We pose keystep recognition as a node classification task, and propose a flexible graph-learning framework for fine-grained keystep recognition that is able to effectively leverage long-term dependencies in egocentric videos. Our approach, termed GLEVR, consists of constructing a graph where each video clip of the egocentric video corresponds to a node. The constructed graphs are sparse and computationally efficient, outperforming existing larger models substantially. We further leverage alignment between egocentric and exocentric videos during training for improved inference on egocentric videos, as well as adding automatic captioning as an additional modality. We consider each clip of each exocentric video (if available) or video captions as additional nodes during training. We examine several strategies to define connections across these nodes. We perform extensive experiments on the Ego-Exo4D dataset and show that our proposed flexible graph-based framework notably outperforms existing methods.

Related papers

• Object-Shot Enhanced Grounding Network for Egocentric Video [60.97916755629796]
  We propose OSGNet, an Object-Shot enhanced Grounding Network for egocentric video.<n>Specifically, we extract object information from videos to enrich video representation.<n>We analyze the frequent shot movements inherent to egocentric videos, leveraging these features to extract the wearer's attention information.
  arXiv  Detail & Related papers  (2025-05-07T09:20:12Z)
• Graph-Based Multimodal and Multi-view Alignment for Keystep Recognition [11.421362760480527]
  We propose a flexible graph-learning framework for fine-grained keystep recognition in egocentric videos.<n>We show that our proposed framework notably outperforms existing methods by more than 12 points in accuracy.<n>We also present a study examining on harnessing several multimodal features, including narrations, depth, and object class labels, on a heterogeneous graph.
  arXiv  Detail & Related papers  (2025-01-07T20:02:55Z)
• VideoSAGE: Video Summarization with Graph Representation Learning [9.21019970479227]
  We propose a graph-based representation learning framework for video summarization. A graph constructed this way aims to capture long-range interactions among video frames, and the sparsity ensures the model trains without hitting the memory and compute bottleneck.
  arXiv  Detail & Related papers  (2024-04-14T15:49:02Z)
• X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization [56.75782714530429]
  We propose a cross-modal adaptation framework, which we call X-MIC. Our pipeline learns to align frozen text embeddings to each egocentric video directly in the shared embedding space. This results in an enhanced alignment of text embeddings to each egocentric video, leading to a significant improvement in cross-dataset generalization.
  arXiv  Detail & Related papers  (2024-03-28T19:45:35Z)
• Edge but not Least: Cross-View Graph Pooling [76.71497833616024]
  This paper presents a cross-view graph pooling (Co-Pooling) method to better exploit crucial graph structure information. Through cross-view interaction, edge-view pooling and node-view pooling seamlessly reinforce each other to learn more informative graph-level representations.
  arXiv  Detail & Related papers  (2021-09-24T08:01:23Z)
• SumGraph: Video Summarization via Recursive Graph Modeling [59.01856443537622]
  We propose graph modeling networks for video summarization, termed SumGraph, to represent a relation graph. We achieve state-of-the-art performance on several benchmarks for video summarization in both supervised and unsupervised manners.
  arXiv  Detail & Related papers  (2020-07-17T08:11:30Z)
• Comprehensive Information Integration Modeling Framework for Video Titling [124.11296128308396]
  We integrate comprehensive sources of information, including the content of consumer-generated videos, the narrative comment sentences supplied by consumers, and the product attributes, in an end-to-end modeling framework. To tackle this issue, the proposed method consists of two processes, i.e., granular-level interaction modeling and abstraction-level story-line summarization. We collect a large-scale dataset accordingly from real-world data in Taobao, a world-leading e-commerce platform.
  arXiv  Detail & Related papers  (2020-06-24T10:38:15Z)
• Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks [150.5425122989146]
  This work proposes a novel attentive graph neural network (AGNN) for zero-shot video object segmentation (ZVOS) AGNN builds a fully connected graph to efficiently represent frames as nodes, and relations between arbitrary frame pairs as edges. Experimental results on three video segmentation datasets show that AGNN sets a new state-of-the-art in each case.
  arXiv  Detail & Related papers  (2020-01-19T10:45:27Z)
• Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data [29.841574293529796]
  We propose Cut-Based Graph Learning Networks (CB-GLNs) for learning video data by discovering complex structures of the video. CB-GLNs represent video data as a graph, with nodes and edges corresponding to frames of the video and their dependencies respectively. We evaluate the proposed method on the two different tasks for video understanding: Video theme classification (Youtube-8M dataset) and Video Question and Answering (TVQA dataset)
  arXiv  Detail & Related papers  (2020-01-17T10:09:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.