UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors

• URL: http://arxiv.org/abs/2603.15975v1
• Date: Mon, 16 Mar 2026 22:44:52 GMT
• Title: UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors
• Abstract summary: We present UMO, a simple yet general unified formulation that casts diverse downstream tasks into compositions of atomic per-frame operations.<n>Specifically, UMO introduces three learnable frame-level meta-operation embeddings to specify per-frame intent and employs lightweight temporal fusion to inject in-context cues into the pretrained backbone.<n>Experiments demonstrate that UMO consistently outperforms task-specific and training-free baselines across a wide range of benchmarks.
• Score: 78.85130555487432
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale foundation models (LFMs) have recently made impressive progress in text-to-motion generation by learning strong generative priors from massive 3D human motion datasets and paired text descriptions. However, how to effectively and efficiently leverage such single-purpose motion LFMs, i.e., text-to-motion synthesis, in more diverse cross-modal and in-context motion generation downstream tasks remains largely unclear. Prior work typically adapts pretrained generative priors to individual downstream tasks in a task-specific manner. In contrast, our goal is to unlock such priors to support a broad spectrum of downstream motion generation tasks within a single unified framework. To bridge this gap, we present UMO, a simple yet general unified formulation that casts diverse downstream tasks into compositions of atomic per-frame operations, enabling in-context adaptation to unlock the generative priors of pretrained DiT-based motion LFMs. Specifically, UMO introduces three learnable frame-level meta-operation embeddings to specify per-frame intent and employs lightweight temporal fusion to inject in-context cues into the pretrained backbone, with negligible runtime overhead compared to the base model. With this design, UMO finetunes the pretrained model, originally limited to text-to-motion generation, to support diverse previously unsupported tasks, including temporal inpainting, text-guided motion editing, text-serialized geometric constraints, and multi-identity reaction generation. Experiments demonstrate that UMO consistently outperforms task-specific and training-free baselines across a wide range of benchmarks, despite using a single unified model. Code and model will be publicly available. Project Page: https://oliver-cong02.github.io/UMO.github.io/

Related papers

• In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics [109.99871326497333]
  In-Context Model Predictive Generation (ICMPG) is a framework that integrates language-model planning with inference-time physical feedback.<n>ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules.
  arXiv  Detail & Related papers  (2026-06-25T12:50:33Z)
• Learning Action Priors for Cross-embodiment Robot Manipulation [46.92215687389683]
  Most Vision-Language-Action (VLA) models build on a Vision-Language Model (VLM) backbone by attaching an action module and optimizing the full policy jointly.<n>This work proposes to pretrain the action module with motion priors before cross-modal VLA alignment.<n>We introduce a two-stage training framework that equips the action module with cross-embodiment temporal motion structure.
  arXiv  Detail & Related papers  (2026-06-24T17:59:56Z)
• MultiAct: Text-to-Motion Generation from Composite Text via Tailored Attention Guidance [54.131431576597514]
  MultiAct is an unpaired, inference-time framework for compositional text-to-motion synthesis.<n>Our method counteracts semantic collapse by adaptively amplifying cross-attention scores associated with underrepresented prompt components.
  arXiv  Detail & Related papers  (2026-05-29T07:14:48Z)
• MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanation [66.66098171359995]
  MotionMERGE is a unified framework that bridges the granularity gap in motion-language models.<n>First, we pioneer the study of fine-grained language-guided motion control, including detailed understanding and localized editing.<n>Second, we design ReasoningAware Granularity-Synergy pre-training, a novel strategy that employs joint supervision for cross-granularity alignment.<n>Third, we curate MotionFineEdit, a large-scale dataset with the first finegrained-temporal corrective instructions and motion-grounded CoT annotations.
  arXiv  Detail & Related papers  (2026-05-18T18:00:04Z)
• LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens [19.167250154665812]
  We propose LLaMo, a framework that extends pretrained large language models through a modality-specific Mixture-of-Transformers architecture.<n>We encode human motion into a causal continuous latent space and maintain the next-token prediction paradigm in the decoder-only backbone.<n>Our experiments demonstrate that LLaMo achieves high-fidelity text-to-motion generation and motion-to-text captioning in general settings.
  arXiv  Detail & Related papers  (2026-02-12T20:02:21Z)
• FrankenMotion: Part-level Human Motion Generation and Composition [41.84042766842064]
  We construct a high-quality motion dataset with atomic, temporally-aware part-level text annotations.<n>Our dataset captures asynchronous and semantically distinct part movements at fine temporal resolution.<n>Based on this dataset, we introduce a diffusion-based part-aware motion generation framework, namely FrankenMotion.
  arXiv  Detail & Related papers  (2026-01-15T23:50:07Z)
• ThinkGen: Generalized Thinking for Visual Generation [97.19923474851987]
  ThinkGen is a think-driven visual generation framework that explicitly leverages Chain-of-Thought (CoT) reasoning in various generation scenarios.<n>We propose a separable GRPO-based training paradigm, alternating reinforcement learning between the MLLM and DiT modules.<n>Experiments demonstrate that ThinkGen achieves robust, state-of-the-art performance across multiple generation benchmarks.
  arXiv  Detail & Related papers  (2025-12-29T16:08:50Z)
• UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models [12.21686773633269]
  Large language models, trained on extensive corpora, successfully unify diverse linguistic tasks within a single generative framework.<n>Such modeling requires task-specific pre-training across modalities and sources, which is costly and limits scalability to unseen tasks.<n>We propose UniVid, a framework that fine-tunes a video diffusion transformer to handle various vision tasks without task-specific modifications.
  arXiv  Detail & Related papers  (2025-09-26T01:43:40Z)
• Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sports [53.637837706712794]
  We propose a Unified Trajectory Generation model, UniTraj, that processes arbitrary trajectories as masked inputs.<n>Specifically, we introduce a Ghost Spatial Masking (GSM) module, embedded within a Transformer encoder, for spatial feature extraction.<n>We benchmark three practical sports datasets, Basketball-U, Football-U, and Soccer-U, for evaluation.
  arXiv  Detail & Related papers  (2024-05-27T22:15:23Z)
• M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation [78.77004913030285]
  M$3$GPT is an advanced $textbfM$ultimodal, $textbfM$ultitask framework for comprehension and generation. We employ discrete vector quantization for multimodal conditional signals, such as text, music and motion/dance, enabling seamless integration into a large language model. M$3$GPT learns to model the connections and synergies among various motion-relevant tasks.
  arXiv  Detail & Related papers  (2024-05-25T15:21:59Z)
• OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers [45.808597624491156]
  We present OMG, a novel framework, which enables compelling motion generation from zero-shot open-vocabulary text prompts. At the pre-training stage, our model improves the generation ability by learning the rich out-of-domain inherent motion traits. At the fine-tuning stage, we introduce motion ControlNet, which incorporates text prompts as conditioning information.
  arXiv  Detail & Related papers  (2023-12-14T14:31:40Z)
• mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections [104.14624185375897]
  mPLUG is a new vision-language foundation model for both cross-modal understanding and generation. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, such as image captioning, image-text retrieval, visual grounding and visual question answering.
  arXiv  Detail & Related papers  (2022-05-24T11:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.