Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Models

• URL: http://arxiv.org/abs/2512.22249v1
• Date: Wed, 24 Dec 2025 03:11:12 GMT
• Title: Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Models
• Abstract summary: Unsupervised human motion segmentation can be effectively achieved using subspace clustering techniques.<n>This paper explores the use of temporal vision semantics (TVS) derived from human motion sequences to enhance subspace clustering performance.<n> Experimental results demonstrate that the proposed method outperforms existing state-of-the-art approaches on four benchmark human motion datasets.
• Score: 22.529947693320864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unsupervised human motion segmentation (HMS) can be effectively achieved using subspace clustering techniques. However, traditional methods overlook the role of temporal semantic exploration in HMS. This paper explores the use of temporal vision semantics (TVS) derived from human motion sequences, leveraging the image-to-text capabilities of a large language model (LLM) to enhance subspace clustering performance. The core idea is to extract textual motion information from consecutive frames via LLM and incorporate this learned information into the subspace clustering framework. The primary challenge lies in learning TVS from human motion sequences using LLM and integrating this information into subspace clustering. To address this, we determine whether consecutive frames depict the same motion by querying the LLM and subsequently learn temporal neighboring information based on its response. We then develop a TVS-integrated subspace clustering approach, incorporating subspace embedding with a temporal regularizer that induces each frame to share similar subspace embeddings with its temporal neighbors. Additionally, segmentation is performed based on subspace embedding with a temporal constraint that induces the grouping of each frame with its temporal neighbors. We also introduce a feedback-enabled framework that continuously optimizes subspace embedding based on the segmentation output. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art approaches on four benchmark human motion datasets.

Related papers

• DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation [53.011370226020695]
  We propose DINOde, an ODE-based framework that aligns CLIP text embeddings with the DINO visual manifold.<n>Our approach employs two complementary components: (i) Semantic Text Flow (STF), which evolves text embeddings toward the DINO manifold through a continuous ODE trajectory, and (ii) Global Context Flow (GCF), which refines the holistic image representation carried by DINO's CLS token.<n>By modeling alignment as a continuous trajectory, DINOde avoids the entanglement inherent in discrete projections and yields more robust cross-modal alignment.
  arXiv  Detail & Related papers  (2026-07-23T14:37:27Z)
• Jointly Learning Structured Representations and Stabilized Affinity for Human Motion Segmentation [7.350724521347576]
  Human Motion Clustering (HMS) aims to partition a video into non-overlapping segments corresponding to different human motions.<n>For videos in the real world, raw frame-level features often violate the Unionof-Subspaces assumption and yield unsatisfactory segmentation performance.<n>We propose Temporal Deep Self-expressive subspace Clustering ( TDSC) which jointly learns temporally consistent structured representations and affinity for accurate and robust HMS.
  arXiv  Detail & Related papers  (2026-05-07T06:48:53Z)
• Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining [86.49790441700195]
  ClusterSTM is a Cluster-Wise Spatio-Temporal Masking strategy for efficient video-language pretraining.<n>We introduce a video-text relevance reconstruction objective that aligns high-level multimodal semantics beyond conventional visual reconstruction.
  arXiv  Detail & Related papers  (2026-03-24T08:48:15Z)
• Learning Spatio-Temporal Feature Representations for Video-Based Gaze Estimation [50.05866669110754]
  Video-based gaze estimation methods aim to capture the inherently temporal dynamics of human eye gaze from multiple image frames.<n>We propose the Spatio-Temporal Gaze Network (ST-Gaze), a model that combines a CNN backbone with dedicated channel attention and self-attention modules.<n>We show that ST-Gaze achieves state-of-the-art performance both with and without person-specific adaptation.
  arXiv  Detail & Related papers  (2025-12-19T15:15:58Z)
• Temporal Rate Reduction Clustering for Human Motion Segmentation [2.902719887963343]
  We propose Temporal Rate Reduction Clustering ($textTR2textC$), which jointly learns structured representations and affinity to segment the sequences of frames in video.<n>We conduct extensive experiments on five benchmark HMS datasets and achieve state-of-the-art performances with different feature extractors.
  arXiv  Detail & Related papers  (2025-06-26T13:35:07Z)
• Cluster-based Video Summarization with Temporal Context Awareness [9.861215740353247]
  TAC-SUM is a novel and efficient training-free approach for video summarization. Our method partitions the input video into temporally consecutive segments with clustering information. The resulting temporal-aware clusters are then utilized to compute the final summary.
  arXiv  Detail & Related papers  (2024-04-06T05:55:14Z)
• Temporally Consistent Referring Video Object Segmentation with Hybrid Memory [98.80249255577304]
  We propose an end-to-end R-VOS paradigm that explicitly models temporal consistency alongside the referring segmentation. Features of frames with automatically generated high-quality reference masks are propagated to segment remaining frames. Extensive experiments demonstrate that our approach enhances temporal consistency by a significant margin.
  arXiv  Detail & Related papers  (2024-03-28T13:32:49Z)
• Anchor-based Multi-view Subspace Clustering with Hierarchical Feature Descent [46.86939432189035]
  We propose Anchor-based Multi-view Subspace Clustering with Hierarchical Feature Descent. Our proposed model consistently outperforms the state-of-the-art techniques.
  arXiv  Detail & Related papers  (2023-10-11T03:29:13Z)
• Spatio-Temporal Branching for Motion Prediction using Motion Increments [55.68088298632865]
  Human motion prediction (HMP) has emerged as a popular research topic due to its diverse applications. Traditional methods rely on hand-crafted features and machine learning techniques. We propose a noveltemporal-temporal branching network using incremental information for HMP.
  arXiv  Detail & Related papers  (2023-08-02T12:04:28Z)
• Hierarchical Spatio-Temporal Representation Learning for Gait Recognition [6.877671230651998]
  Gait recognition is a biometric technique that identifies individuals by their unique walking styles. We propose a hierarchical-temporal representation learning framework for extracting gait features from coarse to fine. Our method outperforms the state-of-the-art while maintaining a reasonable balance between model accuracy and complexity.
  arXiv  Detail & Related papers  (2023-07-19T09:30:00Z)
• SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation [35.063881868130075]
  This paper studies referring video object segmentation (RVOS) by boosting video-level visual-linguistic alignment. We propose Semantic-assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. We conduct extensive experiments on popular RVOS benchmarks, and our method outperforms state-of-the-art competitors on all benchmarks by a remarkable margin.
  arXiv  Detail & Related papers  (2023-05-26T15:13:44Z)
• What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions [55.574102714832456]
  spatial-temporal grounding describes the task of localizing events in space and time. Models for this task are usually trained with human-annotated sentences and bounding box supervision. We combine local representation learning, which focuses on fine-grained spatial information, with a global representation that captures higher-level representations.
  arXiv  Detail & Related papers  (2023-03-29T19:38:23Z)
• Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding [112.3913646778859]
  We propose a simple yet effective video-language modeling framework, S-ViLM. It includes two novel designs, inter-clip spatial grounding and intra-clip temporal grouping, to promote learning region-object alignment and temporal-aware features. S-ViLM surpasses the state-of-the-art methods substantially on four representative downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T22:45:07Z)
• Spatiotemporal k-means [39.98633724527769]
  We propose a twotemporal clustering method called k-means (STk) that is able to analyze multi-scale clusters. We show how STkM can be extended to more complex machine learning tasks, particularly unsupervised region of interest detection and tracking in videos.
  arXiv  Detail & Related papers  (2022-11-10T04:40:31Z)
• Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation [58.74791043631219]
  We propose a novel framework STswinCL that explores the complementary intra- and inter-video relations to boost segmentation performance. We extensively validate our approach on two public surgical video benchmarks, including EndoVis18 Challenge and CaDIS dataset. Experimental results demonstrate the promising performance of our method, which consistently exceeds previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-29T05:52:23Z)
• Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering [10.057155889852174]
  We present a novel approach for unsupervised activity segmentation which uses video frame clustering as a pretext task. We leverage temporal information in videos by employing temporal optimal transport. Our approach performs on par with or better than previous methods, despite having significantly less memory constraints.
  arXiv  Detail & Related papers  (2021-05-27T17:57:37Z)
• Co-Saliency Spatio-Temporal Interaction Network for Person Re-Identification in Videos [85.6430597108455]
  We propose a novel Co-Saliency Spatio-Temporal Interaction Network (CSTNet) for person re-identification in videos. It captures the common salient foreground regions among video frames and explores the spatial-temporal long-range context interdependency from such regions. Multiple spatialtemporal interaction modules within CSTNet are proposed, which exploit the spatial and temporal long-range context interdependencies on such features and spatial-temporal information correlation.
  arXiv  Detail & Related papers  (2020-04-10T10:23:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.