CoMoGen: COntrollable MOtion Dynamics and Interactions with Mask-Guided Video GENeration

• URL: http://arxiv.org/abs/2605.22996v1
• Date: Thu, 21 May 2026 19:51:20 GMT
• Title: CoMoGen: COntrollable MOtion Dynamics and Interactions with Mask-Guided Video GENeration
• Abstract summary: CoMoGen is a controllable video generation framework that generates realistic interactive dynamics from a single binary mask sequence conditioned on an input image.<n>CoMoGen consistently outperforms prior controllable video generation methods and achieves state-of-the-art performance in motion fidelity and perceptual realism.
• Score: 66.81078857385698
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present CoMoGen, a controllable video generation framework that generates realistic interactive dynamics from a single binary mask sequence conditioned on an input image. CoMoGen introduces a lightweight MaskAdapter that encodes binary mask sequences into a latent residual signal, injected into the Multi Modal Diffusion Transformer (MMDiT) model through a cosine-weighted schedule. Unlike the hierarchical coarse-to-fine design of UNet architectures, MMDiT operates as a sequence of uniform transformer blocks, making it difficult to identify which layers are responsible for the motion generation. Therefore, we propose a novel way to determine "Motion Layers" operating in the attention space of MMDiT. We fine-tune the model by using Low-Rank Adaptation (LoRA) to the Motion Layers, without requiring any architecture change in the MMDiT. This selective adaptation enables our method to focus on motion-critical components, yielding reduced computational cost. Despite its simplicity, CoMoGen enables precise subject motion and plausible interactions with surrounding humans, objects, and scenes. Comprehensive experiments on different datasets show that CoMoGen consistently outperforms prior controllable video generation methods and achieves state-of-the-art performance in motion fidelity and perceptual realism. Project page: mericadil.github.io/CoMoGen.

Related papers

• Following Motion for Sequential Modeling in Video Frame Interpolation [22.71397762551783]
  State Space Models (SSMs) have surfaced as a promising architecture in Video Frame Interpolation (VFI)<n>We propose Motion-Guided Mamba for Video Frame Interpolation (MGMVFI), an adaptation of the selective state space model tailored explicitly for VFI.
  arXiv  Detail & Related papers  (2026-08-24T06:43:48Z)
• AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling [60.13338774249153]
  We introduce OmniHuMo, a dataset of over 5,000 hours of motion and 3.2 million sequences with precisely aligned multimodal annotations.<n>We propose AnyMo, a unified multimodal framework combining a Residual FSQ-based motion tokenizer with a scalable masked modeling transformer.<n>Experiments show that AnyMo achieves high-fidelity synthesis while offering flexible control over both spatial and stylistic attributes.
  arXiv  Detail & Related papers  (2026-05-28T07:15:19Z)
• Not All Frames Are Equal: Complexity-Aware Masked Motion Generation via Motion Spectral Descriptors [10.685712212496753]
  Masked generative models have become a strong paradigm for text-to-motion synthesis, but they still treat motion frames too uniformly.<n>We show that current masked motion generators degrade disproportionately on dynamically complex motions.<n>Motivated by this mismatch, we introduce the Motion Spectral Descriptor (MSD), a simple and parameter-free measure of local dynamic complexity.
  arXiv  Detail & Related papers  (2026-03-31T12:18:57Z)
• DiMo: Discrete Diffusion Modeling for Motion Generation and Understanding [25.254783224309488]
  We present DiMo, a discrete diffusion-style framework, which extends masked modeling to text--motion understanding and generation.<n>Unlike GPT-style autoregressive approaches that tokenize motion and decode sequentially, DiMo performs iterative masked token refinement.<n>Experiments on HumanML3D and KIT-ML show strong motion quality and competitive bidirectional understanding.
  arXiv  Detail & Related papers  (2026-02-04T04:01:02Z)
• EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer [64.69014756863331]
  We introduce EchoMotion, a framework designed to model the joint distribution of appearance and human motion.<n>We also propose MVS-RoPE, which offers unified 3D positional encoding for both video and motion tokens.<n>Our findings reveal that explicitly representing human motion is to appearance, significantly boosting the coherence and plausibility of human-centric video generation.
  arXiv  Detail & Related papers  (2025-12-21T17:08:14Z)
• MoSa: Motion Generation with Scalable Autoregressive Modeling [41.75643989500953]
  We introduce MoSa, a novel hierarchical motion generation framework for text-driven 3D human motion generation.<n>MoSa achieves state-of-the-art generation quality and efficiency, outperforming prior methods in both fidelity and speed.<n>MoSa generalizes well to downstream tasks such as motion editing, requiring no additional fine-tuning.
  arXiv  Detail & Related papers  (2025-11-03T03:47:58Z)
• CoMo: Compositional Motion Customization for Text-to-Video Generation [40.446146411270156]
  CoMo is a novel framework for $textbfcompositional motion customization$ in text-to-video generation.<n>It addresses the challenges of motion-merge entanglement and ineffective multi-motion blending.<n>CoMo achieves state-of-the-art performance, significantly advancing the capabilities of controllable video generation.
  arXiv  Detail & Related papers  (2025-10-27T04:57:09Z)
• FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing [56.29102849106382]
  FineMoGen is a diffusion-based motion generation and editing framework. It can synthesize fine-grained motions, with spatial-temporal composition to the user instructions. FineMoGen further enables zero-shot motion editing capabilities with the aid of modern large language models.
  arXiv  Detail & Related papers  (2023-12-22T16:56:02Z)
• Interactive Character Control with Auto-Regressive Motion Diffusion Models [18.727066177880708]
  We propose A-MDM (Auto-regressive Motion Diffusion Model) for real-time motion synthesis. Our conditional diffusion model takes an initial pose as input, and auto-regressively generates successive motion frames conditioned on previous frame. We introduce a suite of techniques for incorporating interactive controls into A-MDM, such as task-oriented sampling, in-painting, and hierarchical reinforcement learning.
  arXiv  Detail & Related papers  (2023-06-01T07:48:34Z)
• MoDi: Unconditional Motion Synthesis from Diverse Data [51.676055380546494]
  We present MoDi, an unconditional generative model that synthesizes diverse motions. Our model is trained in a completely unsupervised setting from a diverse, unstructured and unlabeled motion dataset. We show that despite the lack of any structure in the dataset, the latent space can be semantically clustered.
  arXiv  Detail & Related papers  (2022-06-16T09:06:25Z)
• Unsupervised Motion Representation Learning with Capsule Autoencoders [54.81628825371412]
  Motion Capsule Autoencoder (MCAE) models motion in a two-level hierarchy. MCAE is evaluated on a novel Trajectory20 motion dataset and various real-world skeleton-based human action datasets.
  arXiv  Detail & Related papers  (2021-10-01T16:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.