OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructions

• URL: http://arxiv.org/abs/2512.19159v1
• Date: Mon, 22 Dec 2025 08:55:23 GMT
• Title: OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructions
• Authors: Wendong Bu, Kaihang Pan, Yuze Lin, Jiacheng Li, Kai Shen, Wenqiao Zhang, Juncheng Li, Jun Xiao, Siliang Tang,
• Abstract summary: We propose a unified framework that enables versatile motion generation through interleaved text-motion instructions.<n>Built upon a concise RVQ-VAE and transformer architecture, OmniMoGen supports end-to-end instruction-driven motion generation.<n>Experiments show that OmniMoGen achieves state-of-the-art performance on text-to-motion, motion editing, and AnyContext.
• Score: 50.40879516807197
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have unified diverse linguistic tasks within a single framework, yet such unification remains unexplored in human motion generation. Existing methods are confined to isolated tasks, limiting flexibility for free-form and omni-objective generation. To address this, we propose OmniMoGen, a unified framework that enables versatile motion generation through interleaved text-motion instructions. Built upon a concise RVQ-VAE and transformer architecture, OmniMoGen supports end-to-end instruction-driven motion generation. We construct X2Mo, a large-scale dataset of over 137K interleaved text-motion instructions, and introduce AnyContext, a benchmark for evaluating interleaved motion generation. Experiments show that OmniMoGen achieves state-of-the-art performance on text-to-motion, motion editing, and AnyContext, exhibiting emerging capabilities such as compositional editing, self-reflective generation, and knowledge-informed generation. These results mark a step toward the next intelligent motion generation. Project Page: https://OmniMoGen.github.io/.

Related papers

• IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation [54.36300724708094]
  Assessment and refinement tasks act as crucial bridges to enable bidirectional knowledge flow between understanding and generation.<n>We introduce IRG-MotionLLM, the first model that seamlessly interleaves motion generation, assessment, and refinement to improve generation performance.
  arXiv  Detail & Related papers  (2025-12-11T15:16:06Z)
• GENMO: A GENeralist Model for Human MOtion [64.16188966024542]
  We present GENMO, a unified Generalist Model for Human Motion that bridges motion estimation and generation in a single framework.<n>Our key insight is to reformulate motion estimation as constrained motion generation, where the output motion must precisely satisfy observed conditioning signals.<n>Our novel architecture handles variable-length motions and mixed multimodal conditions (text, audio, video) at different time intervals, offering flexible control.
  arXiv  Detail & Related papers  (2025-05-02T17:59:55Z)
• MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities [36.42160163142448]
  We pioneer MG-MotionLLM, a unified motion-language model for multi-granular motion comprehension and generation.<n>We introduce a comprehensive multi-granularity training scheme by incorporating a set of novel auxiliary tasks.<n>Our MG-MotionLLM achieves superior performance on classical text-to-motion and motion-to-text tasks.
  arXiv  Detail & Related papers  (2025-04-03T10:53:41Z)
• MoTe: Learning Motion-Text Diffusion Model for Multiple Generation Tasks [30.333659816277823]
  We presenttextbfMoTe, a unified multi-modal model that could handle diverse tasks by learning the marginal, conditional, and joint distributions of motion and text simultaneously.<n>MoTe is composed of three components: Motion-Decoder (MED), Text-Decoder (TED), and Moti-on-Text Diffusion Model (MTDM)
  arXiv  Detail & Related papers  (2024-11-29T15:48:24Z)
• KinMo: Kinematic-aware Human Motion Understanding and Generation [6.962697597686156]
  Current human motion synthesis frameworks rely on global action descriptions.<n>A single coarse description, such as run, fails to capture details such as variations in speed, limb positioning, and kinematic dynamics.<n>We introduce KinMo, a unified framework built on a hierarchical describable motion representation.
  arXiv  Detail & Related papers  (2024-11-23T06:50:11Z)
• MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations [85.85596165472663]
  We build MotionBank, which comprises 13 video action datasets, 1.24M motion sequences, and 132.9M frames of natural and diverse human motions. Our MotionBank is beneficial for general motion-related tasks of human motion generation, motion in-context generation, and motion understanding.
  arXiv  Detail & Related papers  (2024-10-17T17:31:24Z)
• Infinite Motion: Extended Motion Generation via Long Text Instructions [51.61117351997808]
  "Infinite Motion" is a novel approach that leverages long text to extended motion generation. Key innovation of our model is its ability to accept arbitrary lengths of text as input. We incorporate the timestamp design for text which allows precise editing of local segments within the generated sequences.
  arXiv  Detail & Related papers  (2024-07-11T12:33:56Z)
• Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs [67.59291068131438]
  Motion-Agent is a conversational framework designed for general human motion generation, editing, and understanding. Motion-Agent employs an open-source pre-trained language model to develop a generative agent, MotionLLM, that bridges the gap between motion and text.
  arXiv  Detail & Related papers  (2024-05-27T09:57:51Z)
• FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing [56.29102849106382]
  FineMoGen is a diffusion-based motion generation and editing framework. It can synthesize fine-grained motions, with spatial-temporal composition to the user instructions. FineMoGen further enables zero-shot motion editing capabilities with the aid of modern large language models.
  arXiv  Detail & Related papers  (2023-12-22T16:56:02Z)
• MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model [35.32967411186489]
  MotionDiffuse is a diffusion model-based text-driven motion generation framework. It excels at modeling complicated data distribution and generating vivid motion sequences. It responds to fine-grained instructions on body parts, and arbitrary-length motion synthesis with time-varied text prompts.
  arXiv  Detail & Related papers  (2022-08-31T17:58:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.