Open-UniMo: Towards Unified Motion-Language Understanding and Generation in the Open World

• URL: http://arxiv.org/abs/2609.14615v1
• Date: Sun, 13 Sep 2026 15:46:41 GMT
• Title: Open-UniMo: Towards Unified Motion-Language Understanding and Generation in the Open World
• Abstract summary: We propose Open-UniMo, a unified Large Motion-Language Model (LMLM) trained on million-scale open-world motion-language data.<n>Open-UniMo promotes modality parity by extending Qwen's vocabulary of about 150K text tokens with 64K motion tokens.<n>We also introduce motion-consistent Chain-of-Thought reasoning as an intermediate representation to bridge language semantics and motion dynamics.
• Score: 31.64682340600284
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unified motion generation and understanding is crucial for embodied AI systems that can both synthesize and interpret human actions in open-world environments. Existing motion-language models often treat motion as an auxiliary modality of a language model, leading to text-dominated representations and limited cross-modal interaction. Moreover, the next-token prediction paradigm is not naturally suited to long motion sequences, where autoregressive generation may accumulate prediction errors. To address these challenges, we propose Open-UniMo, a unified Large Motion-Language Model (LMLM) trained on million-scale open-world motion-language data. Open-UniMo promotes modality parity by extending Qwen's vocabulary of about 150K text tokens with 64K motion tokens, enabling motion and language to share a unified token space. We further introduce motion-consistent Chain-of-Thought reasoning as an intermediate representation to bridge language semantics and motion dynamics. Open-UniMo is trained with a two-stage pipeline, where supervised fine-tuning establishes CoT-guided bidirectional motion-language mapping and Group Relative Policy Optimization (GRPO) improves semantic alignment while mitigating cumulative errors in autoregressive motion-token generation. To support comprehensive evaluation, we propose Open-MoBench, a VLM-guided benchmark for assessing text-to-motion (T2M) generation, motion-to-text (M2T) understanding, and bidirectional consistency. Extensive experiments show that Open-UniMo achieves state-of-the-art performance on both conventional metrics and Open-MoBench. Furthermore, ablation studies reveal that M2T understanding is not primarily limited by motion-token vocabulary size; instead, coupling M2T with the learnable T2M generation path yields stronger cross-modal representations, demonstrating that generation can facilitate understanding in AR-based motion-language modeling.

Related papers

• In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics [109.99871326497333]
  In-Context Model Predictive Generation (ICMPG) is a framework that integrates language-model planning with inference-time physical feedback.<n>ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules.
  arXiv  Detail & Related papers  (2026-06-25T12:50:33Z)
• MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanation [66.66098171359995]
  MotionMERGE is a unified framework that bridges the granularity gap in motion-language models.<n>First, we pioneer the study of fine-grained language-guided motion control, including detailed understanding and localized editing.<n>Second, we design ReasoningAware Granularity-Synergy pre-training, a novel strategy that employs joint supervision for cross-granularity alignment.<n>Third, we curate MotionFineEdit, a large-scale dataset with the first finegrained-temporal corrective instructions and motion-grounded CoT annotations.
  arXiv  Detail & Related papers  (2026-05-18T18:00:04Z)
• LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens [19.167250154665812]
  We propose LLaMo, a framework that extends pretrained large language models through a modality-specific Mixture-of-Transformers architecture.<n>We encode human motion into a causal continuous latent space and maintain the next-token prediction paradigm in the decoder-only backbone.<n>Our experiments demonstrate that LLaMo achieves high-fidelity text-to-motion generation and motion-to-text captioning in general settings.
  arXiv  Detail & Related papers  (2026-02-12T20:02:21Z)
• DiMo: Discrete Diffusion Modeling for Motion Generation and Understanding [25.254783224309488]
  We present DiMo, a discrete diffusion-style framework, which extends masked modeling to text--motion understanding and generation.<n>Unlike GPT-style autoregressive approaches that tokenize motion and decode sequentially, DiMo performs iterative masked token refinement.<n>Experiments on HumanML3D and KIT-ML show strong motion quality and competitive bidirectional understanding.
  arXiv  Detail & Related papers  (2026-02-04T04:01:02Z)
• OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructions [50.40879516807197]
  We propose a unified framework that enables versatile motion generation through interleaved text-motion instructions.<n>Built upon a concise RVQ-VAE and transformer architecture, OmniMoGen supports end-to-end instruction-driven motion generation.<n>Experiments show that OmniMoGen achieves state-of-the-art performance on text-to-motion, motion editing, and AnyContext.
  arXiv  Detail & Related papers  (2025-12-22T08:55:23Z)
• DEFT-LLM: Disentangled Expert Feature Tuning for Micro-Expression Recognition [16.903294278064667]
  We propose DEFT-LLM, which achieves semantic alignment by multi-expert disentanglement.<n>We first introduce Uni-MER, a motion-driven instruction designed to align text with local facial motion.<n>We then design an architecture with three experts to decouple facial dynamics into independent representations.
  arXiv  Detail & Related papers  (2025-11-14T04:21:24Z)
• MoGIC: Boosting Motion Generation via Intention Understanding and Visual Context [16.15137439724085]
  MoGIC is a framework that integrates intention modeling and visual priors into multimodal motion synthesis.<n>We introduce a mixture-of-attention mechanism with adaptive scope to enable effective local alignment between conditional tokens and motion subsequences.<n>Experiments show it reduces FID by 38.6% on HumanML3D and 34.6% on Mo440H, surpasses LLM-based methods in motion captioning with a lightweight text head.
  arXiv  Detail & Related papers  (2025-10-03T04:54:39Z)
• MotionGPT3: Human Motion as a Second Modality [28.616340011811843]
  MotionGPT3 is a bimodal motion-language model for both understanding and generation.<n>A dual-stream Transformer with shared attention preserves modality-specific routes while enabling controlled, bidirectional information flow.<n>Experiments show that MotionGPT3 achieves 2x faster convergence in training loss and up to 4x faster convergence in validation.
  arXiv  Detail & Related papers  (2025-06-30T17:42:22Z)
• ReAlign: Bilingual Text-to-Motion Generation via Step-Aware Reward-Guided Alignment [48.894439350114396]
  We propose a novel bilingual human motion dataset, BiHumanML3D, which establishes a crucial benchmark for bilingual text-to-motion generation models.<n>We also propose a Bilingual Motion Diffusion model (BiMD), which leverages cross-lingual aligned representations to capture semantics.<n>We show that our approach significantly improves text-motion alignment and motion quality compared to existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-08T06:19:18Z)
• MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding [76.30210465222218]
  MotionGPT-2 is a unified Large Motion-Language Model (LMLMLM) It supports multimodal control conditions through pre-trained Large Language Models (LLMs) It is highly adaptable to the challenging 3D holistic motion generation task.
  arXiv  Detail & Related papers  (2024-10-29T05:25:34Z)
• Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs [67.59291068131438]
  Motion-Agent is a conversational framework designed for general human motion generation, editing, and understanding. Motion-Agent employs an open-source pre-trained language model to develop a generative agent, MotionLLM, that bridges the gap between motion and text.
  arXiv  Detail & Related papers  (2024-05-27T09:57:51Z)
• DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion [70.33381660741861]
  We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions. We show that our DiverseMotion achieves the state-of-the-art motion quality and competitive motion diversity.
  arXiv  Detail & Related papers  (2023-09-04T05:43:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.