Beyond Pairwise Relations: Dynamic Manipulation Hypergraphs for Vision-Based Human Activity Recognition

• URL: http://arxiv.org/abs/2607.14350v2
• Date: Mon, 20 Jul 2026 09:32:20 GMT
• Title: Beyond Pairwise Relations: Dynamic Manipulation Hypergraphs for Vision-Based Human Activity Recognition
• Abstract summary: Fine-grained manipulation recognition requires modeling evolving relations among hands, objects, tools, and supporting surfaces.<n>We propose a dynamic manipulation hypergraph framework that represents multi-entity configurations as higher-order relational units.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-grained manipulation recognition requires modeling evolving relations among hands, objects, tools, and supporting surfaces. Conventional graph-based methods use pairwise edges that can fragment a coordinated event into disconnected binary relations. We propose a dynamic manipulation hypergraph framework that represents multi-entity configurations as higher-order relational units. At each temporal step, relevant entities are encoded using appearance, spatial, motion, and semantic-role features. Hyperedge candidates are instantiated and ranked using proximity, contact, and motion-coupling predicates. A hypergraph reasoning network performs node-to-hyperedge and hyperedge-to-node message passing, followed by temporal attention over the evolving interaction structure. The framework provides class-agnostic hyperedge-importance scores that identify entity configurations and temporal intervals emphasized by the model without treating them as causal explanations. Quantitative evaluation is conducted on EPIC-KITCHENS-100/VISOR and Assembly101 under an annotation-assisted entity-localization protocol. Video-only and entity-based methods provide contextual comparisons, while a matched pairwise graph and a static hypergraph serve as the principal controlled baselines because they use identical entity inputs and comparable relational settings. The proposed method improves HO-F1 over the matched pairwise graph by 6.9 percentage points on EPIC-KITCHENS-100/VISOR and 9.5 points on Assembly101, and exceeds the static hypergraph by 4.4 and 5.8 points, respectively. Qualitative analysis on ARCTIC further shows correspondence between highly ranked hyperedges and contact-rich manipulation intervals. These results demonstrate the value of time-varying higher-order relational modeling for fine-grained manipulation activity recognition.

Related papers

• Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis [55.78822230833247]
  We present Inter-X++, a comprehensive and large-scale benchmark designed to empower versatile HHI analysis.<n>Inter-X++ provides 11,388 high-fidelity interaction sequences and over 8.1M frames, featuring precise whole-body movements and detailed finger articulations.<n>We propose OpenHHI, a single and unified HHI representation and modeling framework that jointly optimize interaction reconstruction and semantic understanding.
  arXiv  Detail & Related papers  (2026-08-20T17:51:48Z)
• Edge-Centric Relational Reasoning for 3D Scene Graph Prediction [74.19580969696898]
  3D scene graph prediction aims to abstract complex 3D environments into structured graphs consisting of objects and their pairwise relationships.<n>Existing approaches typically adopt object-centric graph neural networks, where relation edge features are iteratively updated by aggregating messages from connected object nodes.<n>We propose a Link-guided Edge-centric relational reasoning framework with Object-aware fusion.
  arXiv  Detail & Related papers  (2025-11-19T09:53:56Z)
• SEP-GCN: Leveraging Similar Edge Pairs with Temporal and Spatial Contexts for Location-Based Recommender Systems [0.0]
  We propose SEP-GCN, a novel graph-based recommendation framework that learns from pairs of contextually similar interaction edges.<n>By identifying edge pairs that occur within similar temporal windows or geographic proximity, SEP-GCN augments the user-item graph with contextual similarity links.<n> Experiments on benchmark data sets show that SEP-GCN consistently outperforms strong baselines in both predictive accuracy and robustness.
  arXiv  Detail & Related papers  (2025-06-19T03:48:30Z)
• SoftHGNN: Soft Hypergraph Neural Networks for General Visual Recognition [19.88933136559363]
  Hypergraphs extend conventional graphs by modeling high-order interactions.<n>Existing hypergraph neural networks typically rely on static and hard hyperedge assignments.<n>We present Soft Hypergraph Neural Networks (SoftHGNNs), which extend the methodology of hypergraph computation.
  arXiv  Detail & Related papers  (2025-05-21T10:01:30Z)
• Deep Representation Learning for Forecasting Recursive and Multi-Relational Events in Temporal Networks [12.142292322071299]
  This work addresses the problem of forecasting higher-order interaction events that can be multi-relational and recursive.<n>The proposed model, textitRelational Recursive Hyperedge Temporal Point Process (RRHyperTPP), uses an encoder that learns a dynamic node representation based on the historical interaction patterns.<n>We develop a noise contrastive estimation method to learn the parameters of our model, and we have experimentally shown that our models perform better than previous state-of-the-art methods for interaction forecasting.
  arXiv  Detail & Related papers  (2024-04-27T15:46:54Z)
• Hypergraph Transformer for Semi-Supervised Classification [50.92027313775934]
  We propose a novel hypergraph learning framework, HyperGraph Transformer (HyperGT) HyperGT uses a Transformer-based neural network architecture to effectively consider global correlations among all nodes and hyperedges. It achieves comprehensive hypergraph representation learning by effectively incorporating global interactions while preserving local connectivity patterns.
  arXiv  Detail & Related papers  (2023-12-18T17:50:52Z)
• From Hypergraph Energy Functions to Hypergraph Neural Networks [94.88564151540459]
  We present an expressive family of parameterized, hypergraph-regularized energy functions. We then demonstrate how minimizers of these energies effectively serve as node embeddings. We draw parallels between the proposed bilevel hypergraph optimization, and existing GNN architectures in common use.
  arXiv  Detail & Related papers  (2023-06-16T04:40:59Z)
• HIORE: Leveraging High-order Interactions for Unified Entity Relation Extraction [85.80317530027212]
  We propose HIORE, a new method for unified entity relation extraction. The key insight is to leverage the complex association among word pairs, which contains richer information than the first-order word-by-word interactions. Experiments show that HIORE achieves the state-of-the-art performance on relation extraction and an improvement of 1.11.8 F1 points over the prior best unified model.
  arXiv  Detail & Related papers  (2023-05-07T14:57:42Z)
• Multi-Task Edge Prediction in Temporally-Dynamic Video Graphs [16.121140184388786]
  We propose MTD-GNN, a graph network for predicting temporally-dynamic edges for multiple types of relations. We show that modeling multiple relations in our temporal-dynamic graph network can be mutually beneficial.
  arXiv  Detail & Related papers  (2022-12-06T10:41:00Z)
• Modelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence Learning [53.74240452117145]
  This paper presents a self-supervised method for learning reliable visual correspondence from unlabeled videos. We formulate the correspondence as finding paths in a joint space-time graph, where nodes are grid patches sampled from frames, and are linked by two types of edges. Our learned representation outperforms the state-of-the-art self-supervised methods on a variety of visual tasks.
  arXiv  Detail & Related papers  (2021-09-28T05:40:01Z)
• Learning over Families of Sets -- Hypergraph Representation Learning for Higher Order Tasks [12.28143554382742]
  We develop a hypergraph neural network to learn provably expressive representations of variable sized hyperedges. We evaluate performance on multiple real-world hypergraph datasets and demonstrate consistent, significant improvement in accuracy, over state-of-the-art models.
  arXiv  Detail & Related papers  (2021-01-19T18:37:50Z)
• ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection [101.56529337489417]
  We consider the problem of Human-Object Interaction (HOI) Detection, which aims to locate and recognize HOI instances in the form of human, action, object> in images. We argue that multi-level consistencies among objects, actions and interactions are strong cues for generating semantic representations of rare or previously unseen HOIs. Our model takes visual features of candidate human-object pairs and word embeddings of HOI labels as inputs, maps them into visual-semantic joint embedding space and obtains detection results by measuring their similarities.
  arXiv  Detail & Related papers  (2020-08-14T09:11:18Z)
• A Graph-based Interactive Reasoning for Human-Object Interaction Detection [71.50535113279551]
  We present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs. We construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Our framework is end-to-end trainable and free from costly annotations like human pose.
  arXiv  Detail & Related papers  (2020-07-14T09:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.