Fine-grained Motion Retrieval via Joint-Angle Motion Images and Token-Patch Late Interaction

• URL: http://arxiv.org/abs/2603.09930v1
• Date: Tue, 10 Mar 2026 17:26:42 GMT
• Title: Fine-grained Motion Retrieval via Joint-Angle Motion Images and Token-Patch Late Interaction
• Abstract summary: Text-motion retrieval aims to learn a semantically aligned latent space between natural language descriptions and 3D human motion skeleton sequences.<n>Most existing methods use a dual-encoder framework that compresses motion and text into global embeddings.<n>We propose an interpretable, joint-angle-based motion representation that maps joint-level local features into a structured pseudo-image.
• Score: 8.282478908772534
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-motion retrieval aims to learn a semantically aligned latent space between natural language descriptions and 3D human motion skeleton sequences, enabling bidirectional search across the two modalities. Most existing methods use a dual-encoder framework that compresses motion and text into global embeddings, discarding fine-grained local correspondences, and thus reducing accuracy. Additionally, these global-embedding methods offer limited interpretability of the retrieval results. To overcome these limitations, we propose an interpretable, joint-angle-based motion representation that maps joint-level local features into a structured pseudo-image, compatible with pre-trained Vision Transformers. For text-to-motion retrieval, we employ MaxSim, a token-wise late interaction mechanism, and enhance it with Masked Language Modeling regularization to foster robust, interpretable text-motion alignment. Extensive experiments on HumanML3D and KIT-ML show that our method outperforms state-of-the-art text-motion retrieval approaches while offering interpretable fine-grained correspondences between text and motion. The code is available in the supplementary material.

Related papers

• Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures [58.6162288735234]
  We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent.<n>Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript.<n>On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline.
  arXiv  Detail & Related papers  (2026-05-28T22:01:11Z)
• Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor Learning [56.6025512458557]
  Motion-language retrieval aims to bridge the semantic gap between natural language and human motion.<n>Existing approaches predominantly focus on aligning entire motion sequences with global textual representations.<n>We propose a novel Pyramidal Shapley-Taylor (PST) learning framework for fine-grained motion-language retrieval.
  arXiv  Detail & Related papers  (2026-01-29T16:00:12Z)
• SegMo: Segment-aligned Text to 3D Human Motion Generation [8.289973143166062]
  SegMo is a novel text-conditioned human Motion generation framework.<n>It decomposes complex textual descriptions into temporally ordered phrases, each representing a simple atomic action.<n>It partitions motion sequences into corresponding motion segments, and aligns text and motion segments with contrastive learning.
  arXiv  Detail & Related papers  (2025-12-24T15:26:11Z)
• ReAlign: Bilingual Text-to-Motion Generation via Step-Aware Reward-Guided Alignment [48.894439350114396]
  We propose a novel bilingual human motion dataset, BiHumanML3D, which establishes a crucial benchmark for bilingual text-to-motion generation models.<n>We also propose a Bilingual Motion Diffusion model (BiMD), which leverages cross-lingual aligned representations to capture semantics.<n>We show that our approach significantly improves text-motion alignment and motion quality compared to existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-08T06:19:18Z)
• GHOST: Grounded Human Motion Generation with Open Vocabulary Scene-and-Text Contexts [48.28000728061778]
  We propose a method that integrates an open vocabulary scene encoder into the architecture, establishing a robust connection between text and scene. Our methodology achieves up to a 30% reduction in the goal object distance metric compared to the prior state-of-the-art baseline model.
  arXiv  Detail & Related papers  (2024-04-08T18:24:12Z)
• SemanticBoost: Elevating Motion Generation with Augmented Textual Cues [73.83255805408126]
  Our framework comprises a Semantic Enhancement module and a Context-Attuned Motion Denoiser (CAMD) The CAMD approach provides an all-encompassing solution for generating high-quality, semantically consistent motion sequences. Our experimental results demonstrate that SemanticBoost, as a diffusion-based method, outperforms auto-regressive-based techniques.
  arXiv  Detail & Related papers  (2023-10-31T09:58:11Z)
• Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Model [11.873294782380984]
  We propose a fine-grained method for generating high-quality, conditional human motion sequences supporting precise text description. Our approach consists of two key components: 1) a linguistics-structure assisted module that constructs accurate and complete language feature to fully utilize text information; and 2) a context-aware progressive reasoning module that learns neighborhood and overall semantic linguistics features from shallow and deep graph neural networks to achieve a multi-step inference.
  arXiv  Detail & Related papers  (2023-09-12T14:43:47Z)
• AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism [24.049207982022214]
  We propose textbftT2M, a two-stage method with multi-perspective attention mechanism. Our method outperforms the current state-of-the-art in terms of qualitative and quantitative evaluation.
  arXiv  Detail & Related papers  (2023-09-02T02:18:17Z)
• Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language [4.86658723641864]
  We propose a novel text-to-motion retrieval task, which aims at retrieving relevant motions based on a specified natural description. Inspired by the recent progress in text-to-image/video matching, we experiment with two widely-adopted metric-learning loss functions.
  arXiv  Detail & Related papers  (2023-05-25T08:32:41Z)
• TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts [20.336481832461168]
  Inspired by the strong ties between vision and language, our paper aims to explore the generation of 3D human full-body motions from texts. We propose the use of motion token, a discrete and compact motion representation. Our approach is flexible, could be used for both text2motion and motion2text tasks.
  arXiv  Detail & Related papers  (2022-07-04T19:52:18Z)
• Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation [56.41614987789537]
  Text-based video segmentation aims to segment the target object in a video based on a describing sentence. We propose a method to fuse and align appearance, motion, and linguistic features to achieve accurate segmentation.
  arXiv  Detail & Related papers  (2022-04-06T02:42:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.