Text-Derived Relational Graph-Enhanced Network for Skeleton-Based Action Segmentation

• URL: http://arxiv.org/abs/2503.15126v1
• Date: Wed, 19 Mar 2025 11:38:14 GMT
• Title: Text-Derived Relational Graph-Enhanced Network for Skeleton-Based Action Segmentation
• Authors: Haoyu Ji, Bowen Chen, Weihong Ren, Wenze Huang, Zhihao Yang, Zhiyong Wang, Honghai Liu,
• Abstract summary: We propose a Text-Derived Graph Network (TRG-Net) to enhance both modeling and supervision.<n>For modeling, the Dynamic Spatio-Temporal Fusion Modeling (D) method incorporates Text-Derived Joint Graphs (JGT) with channel adaptation.<n>For supervision, the Absolute-Relative Inter-Class Supervision (ARIS) method employs contrastive learning between action features and text embeddings to regularize the absolute class.
• Score: 14.707224594220264
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Skeleton-based Temporal Action Segmentation (STAS) aims to segment and recognize various actions from long, untrimmed sequences of human skeletal movements. Current STAS methods typically employ spatio-temporal modeling to establish dependencies among joints as well as frames, and utilize one-hot encoding with cross-entropy loss for frame-wise classification supervision. However, these methods overlook the intrinsic correlations among joints and actions within skeletal features, leading to a limited understanding of human movements. To address this, we propose a Text-Derived Relational Graph-Enhanced Network (TRG-Net) that leverages prior graphs generated by Large Language Models (LLM) to enhance both modeling and supervision. For modeling, the Dynamic Spatio-Temporal Fusion Modeling (DSFM) method incorporates Text-Derived Joint Graphs (TJG) with channel- and frame-level dynamic adaptation to effectively model spatial relations, while integrating spatio-temporal core features during temporal modeling. For supervision, the Absolute-Relative Inter-Class Supervision (ARIS) method employs contrastive learning between action features and text embeddings to regularize the absolute class distributions, and utilizes Text-Derived Action Graphs (TAG) to capture the relative inter-class relationships among action features. Additionally, we propose a Spatial-Aware Enhancement Processing (SAEP) method, which incorporates random joint occlusion and axial rotation to enhance spatial generalization. Performance evaluations on four public datasets demonstrate that TRG-Net achieves state-of-the-art results.

Related papers

• DynSTG-Mamba: Dynamic Spatio-Temporal Graph Mamba with Cross-Graph Knowledge Distillation for Gait Disorders Recognition [1.7519167857253402]
  DynTG-Mamba is a novel framework that combines DF-STGNN and STG-Mamba to enhance motion modeling. DF-STGNN incorporates a dynamic spatial filter that adaptively adjusts between skeletal joints and temporal interactions. STG-Mamba, an extension of Mamba, ensures a continuous propagation of states while reducing computational costs.
  arXiv  Detail & Related papers  (2025-03-17T13:26:47Z)
• Community-Aware Temporal Walks: Parameter-Free Representation Learning on Continuous-Time Dynamic Graphs [3.833708891059351]
  Community-aware Temporal Walks (CTWalks) is a novel framework for representation learning on continuous-time dynamic graphs.<n>CTWalks integrates a community-based parameter-free temporal walk sampling mechanism, an anonymization strategy enriched with community labels, and an encoding process.<n> Experiments on benchmark datasets demonstrate that CTWalks outperforms established methods in temporal link prediction tasks.
  arXiv  Detail & Related papers  (2025-01-21T04:16:46Z)
• Multi-Scale Spatial-Temporal Self-Attention Graph Convolutional Networks for Skeleton-based Action Recognition [0.0]
  In this paper, we propose self-attention GCN hybrid model, Multi-Scale Spatial-Temporal self-attention (MSST)-GCN. We utilize spatial self-attention module with adaptive topology to understand intra-frame interactions within a frame among different body parts, and temporal self-attention module to examine correlations between frames of a node.
  arXiv  Detail & Related papers  (2024-04-03T10:25:45Z)
• Learning Mutual Excitation for Hand-to-Hand and Human-to-Human Interaction Recognition [21.007782102151282]
  We propose a mutual excitation graph convolutional network (me-GCN) by stacking mutual excitation graph convolution layers.<n>Me-GC learns mutual information in each layer and each stage of graph convolution operations.<n>Our proposed me-GC outperforms state-of-the-art GCN-based and Transformer-based methods.
  arXiv  Detail & Related papers  (2024-02-04T10:00:00Z)
• A Decoupled Spatio-Temporal Framework for Skeleton-based Action Segmentation [89.86345494602642]
  Existing methods are limited in weak-temporal modeling capability. We propose a Decoupled Scoupled Framework (DeST) to address the issues. DeST significantly outperforms current state-of-the-art methods with less computational complexity.
  arXiv  Detail & Related papers  (2023-12-10T09:11:39Z)
• DG-STGCN: Dynamic Spatial-Temporal Modeling for Skeleton-based Action Recognition [77.87404524458809]
  We propose a new framework for skeleton-based action recognition, namely Dynamic Group Spatio-Temporal GCN (DG-STGCN) It consists of two modules, DG-GCN and DG-TCN, respectively, for spatial and temporal modeling. DG-STGCN consistently outperforms state-of-the-art methods, often by a notable margin.
  arXiv  Detail & Related papers  (2022-10-12T03:17:37Z)
• Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition [88.34182299496074]
  Action labels are only available on a source dataset, but unavailable on a target dataset in the training stage. We utilize a self-supervision scheme to reduce the domain shift between two skeleton-based action datasets. By segmenting and permuting temporal segments or human body parts, we design two self-supervised learning classification tasks.
  arXiv  Detail & Related papers  (2022-07-17T07:05:39Z)
• SpatioTemporal Focus for Skeleton-based Action Recognition [66.8571926307011]
  Graph convolutional networks (GCNs) are widely adopted in skeleton-based action recognition. We argue that the performance of recent proposed skeleton-based action recognition methods is limited by the following factors. Inspired by the recent attention mechanism, we propose a multi-grain contextual focus module, termed MCF, to capture the action associated relation information.
  arXiv  Detail & Related papers  (2022-03-31T02:45:24Z)
• Modeling long-term interactions to enhance action recognition [81.09859029964323]
  We propose a new approach to under-stand actions in egocentric videos that exploits the semantics of object interactions at both frame and temporal levels. We use a region-based approach that takes as input a primary region roughly corresponding to the user hands and a set of secondary regions potentially corresponding to the interacting objects. The proposed approach outperforms the state-of-the-art in terms of action recognition on standard benchmarks.
  arXiv  Detail & Related papers  (2021-04-23T10:08:15Z)
• On the spatial attention in Spatio-Temporal Graph Convolutional Networks for skeleton-based human action recognition [97.14064057840089]
  Graphal networks (GCNs) promising performance in skeleton-based human action recognition by modeling a sequence of skeletons as a graph. Most of the recently proposed G-temporal-based methods improve the performance by learning the graph structure at each layer of the network.
  arXiv  Detail & Related papers  (2020-11-07T19:03:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.