ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation

• URL: http://arxiv.org/abs/2609.08365v1
• Date: Tue, 08 Sep 2026 07:33:02 GMT
• Title: ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation
• Abstract summary: Text-to-motion (T2M) generation maps natural language to human joint movements, aiding gaming, VR, and robotics.<n>Retrieval-Augmented Text-to-Motion (RAG-T2M) improves generation on complex descriptions by conditioning on retrieved motion-text pairs.<n>However, existing RAG-T2M models face two challenges: coarse-grained retrieval and fusion mechanisms overlook the hierarchical, spatial-temporal topology of human motion, and a representation gap exists because retrieved evidence resides in a semantic space separate from the generator's latents.<n>We present ReMoMask, a structure
• Score: 41.21060459114798
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Text-to-motion (T2M) generation maps natural language to human joint movements, aiding gaming, VR, and robotics. Retrieval-Augmented Text-to-Motion (RAG-T2M) improves generation on complex descriptions by conditioning on retrieved motion-text pairs. However, existing RAG-T2M models face two challenges: coarse-grained retrieval and fusion mechanisms overlook the hierarchical, spatial-temporal topology of human motion, and a representation gap exists because retrieved evidence resides in a semantic space separate from the generator's latents. To address the first, we present ReMoMask, a structure-aware RAG framework coupling Hierarchical Bidirectional Momentum (HBM) contrastive learning to align global and part-level features with text; Semantic Spatial-Temporal Attention (SSTA) for topology-aware fusion; and Topology Structured Masking (TSM) to force robust part-level grounding via adaptive masking. To address the second, we introduce ReMoMask-2, which rebuilds the retrieval database directly within the generator's pre-quantization latent space and aligns text queries via a distilled lightweight projector, allowing the generator to directly consume the retrieved motion's semantic content. Extensive experiments on HumanML3D, KIT-ML, and SnapMoGen demonstrate our retriever achieves state-of-the-art accuracy, while ReMoMask-2 attains the lowest FID on KIT-ML and SnapMoGen; notably, its single mask-transformer stage surpasses ReMoMask's full two-stage pipeline and delivers the fastest inference.

Related papers

• Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation [50.92925099642139]
  Diffusion Transformers (DiTs) have become the dominant paradigm for high-fidelity video generation, yet their ability to perform high-level semantic planning remains limited.<n>We propose BiVidGen, a hybrid framework where an MLLM first generates semantic visual tokens and a DiT renders videos conditioned on both text and these tokens via multi-layer cross-attention.
  arXiv  Detail & Related papers  (2026-08-14T07:40:41Z)
• CoMoGen: COntrollable MOtion Dynamics and Interactions with Mask-Guided Video GENeration [66.81078857385698]
  CoMoGen is a controllable video generation framework that generates realistic interactive dynamics from a single binary mask sequence conditioned on an input image.<n>CoMoGen consistently outperforms prior controllable video generation methods and achieves state-of-the-art performance in motion fidelity and perceptual realism.
  arXiv  Detail & Related papers  (2026-05-21T19:51:20Z)
• Temporal Consistency-Aware Text-to-Motion Generation [41.71400323450202]
  We propose TCA-T2M, a framework for temporal consistency-aware T2M generation.<n>Our approach introduces a temporal consistency-aware spatial VQ-VAE for cross-sequence temporal alignment.<n> Experiments on HumanML3D and KIT-ML benchmarks demonstrate that TCA-T2M achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-20T08:17:01Z)
• LUMA: Low-Dimension Unified Motion Alignment with Dual-Path Anchoring for Text-to-Motion Diffusion Model [18.564067196226436]
  We propose a text-to-motion diffusion model that incorporates dual-path anchoring to enhance semantic alignment.<n>LUMA achieves state-of-the-art performance, with FID scores of 0.035 and 0.123, respectively.
  arXiv  Detail & Related papers  (2025-09-29T17:58:28Z)
• ReMoMask: Retrieval-Augmented Masked Motion Generation [13.42229775333369]
  Text-to-Motion (T2M) generation aims to synthesize realistic and semantically aligned human motion sequences from natural language descriptions.<n>We propose ReMoMask, a unified framework integrating three key innovations.<n>A Bidirectional Momentum Text-Motion Model decouples negative sample scale from batch size via momentum queues, substantially improving cross-modal retrieval precision.<n>A Semantic Spatio-temporal Attention mechanism enforces biomechanical constraints during part-level fusion to eliminate asynchronous artifacts.
  arXiv  Detail & Related papers  (2025-08-04T16:56:35Z)
• MOGO: Residual Quantized Hierarchical Causal Transformer for High-Quality and Real-Time 3D Human Motion Generation [3.6669020073583756]
  MOGO is a novel autoregressive framework tailored for efficient and real-time 3D motion generation.<n>MoGO comprises two key components: MoSA-VQ, a motion scale-adaptive residual vector quantization module, and RQHC-Transformer, a residual quantized hierarchical causal transformer.<n>To enhance semantic fidelity, we introduce a text condition alignment mechanism that improves motion decoding under textual control.
  arXiv  Detail & Related papers  (2025-06-06T10:26:54Z)
• ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer [58.49950218437718]
  We present ReCoM, an efficient framework for generating high-fidelity and generalizable human body motions synchronized with speech.<n>The core innovation lies in the Recurrent Embedded Transformer (RET), which integrates Dynamic Embedding Regularization (DER) into a Vision Transformer (ViT) core architecture.<n>To enhance model robustness, we incorporate the proposed DER strategy, which equips the model with dual capabilities of noise resistance and cross-domain generalization.
  arXiv  Detail & Related papers  (2025-03-27T16:39:40Z)
• Fg-T2M++: LLMs-Augmented Fine-Grained Text Driven Human Motion Generation [19.094098673523263]
  We propose a novel framework for fine-grained text-driven human motion generation.<n>Fg-T2M++ consists of: (1) an LLMs semantic parsing module to extract body part descriptions and semantics from text, (2) a hyperbolic text representation module to encode relational information between text units, and (3) a multi-modal fusion module to hierarchically fuse text and motion features.
  arXiv  Detail & Related papers  (2025-02-08T11:38:12Z)
• Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation [52.337472185022136]
  We consider the task of Image-to-Video (I2V) generation, which involves transforming static images into realistic video sequences based on a textual description.<n>We propose a two-stage compositional framework that decomposes I2V generation into: (i) An explicit intermediate representation generation stage, followed by (ii) A video generation stage that is conditioned on this representation.<n>We evaluate our method on challenging benchmarks with multi-object and high-motion scenarios and empirically demonstrate that the proposed method achieves state-of-the-art consistency.
  arXiv  Detail & Related papers  (2025-01-06T14:49:26Z)
• MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation [44.74056930805525]
  We introduce a novel Masked Diffusion Transformer for co-speech gesture generation, referred to as MDT-A2G. This model employs a mask modeling scheme specifically designed to strengthen temporal relation learning among sequence gestures. Experimental results demonstrate that MDT-A2G excels in gesture generation, boasting a learning speed that is over 6$times$ faster than traditional diffusion transformers.
  arXiv  Detail & Related papers  (2024-08-06T17:29:01Z)
• Mask Attention Networks: Rethinking and Strengthen Transformer [70.95528238937861]
  Transformer is an attention-based neural network, which consists of two sublayers, Self-Attention Network (SAN) and Feed-Forward Network (FFN)
  arXiv  Detail & Related papers  (2021-03-25T04:07:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.