MaskControl: Spatio-Temporal Control for Masked Motion Synthesis

• URL: http://arxiv.org/abs/2410.10780v3
• Date: Fri, 25 Jul 2025 11:24:03 GMT
• Title: MaskControl: Spatio-Temporal Control for Masked Motion Synthesis
• Abstract summary: We propose MaskControl, the first approach to introduce controllability to the generative masked motion model.<n>First, textitLogits Regularizer implicitly perturbs logits at training time to align the distribution of motion tokens with the controlled joint positions.<n>Second, textitLogit optimization explicitly reshapes the token distribution that forces the generated motion to accurately align with the controlled joint positions.
• Score: 38.16884934336603
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in motion diffusion models have enabled spatially controllable text-to-motion generation. However, these models struggle to achieve high-precision control while maintaining high-quality motion generation. To address these challenges, we propose MaskControl, the first approach to introduce controllability to the generative masked motion model. Our approach introduces two key innovations. First, \textit{Logits Regularizer} implicitly perturbs logits at training time to align the distribution of motion tokens with the controlled joint positions, while regularizing the categorical token prediction to ensure high-fidelity generation. Second, \textit{Logit Optimization} explicitly optimizes the predicted logits during inference time, directly reshaping the token distribution that forces the generated motion to accurately align with the controlled joint positions. Moreover, we introduce \textit{Differentiable Expectation Sampling (DES)} to combat the non-differential distribution sampling process encountered by logits regularizer and optimization. Extensive experiments demonstrate that MaskControl outperforms state-of-the-art methods, achieving superior motion quality (FID decreases by ~77\%) and higher control precision (average error 0.91 vs. 1.08). Additionally, MaskControl enables diverse applications, including any-joint-any-frame control, body-part timeline control, and zero-shot objective control. Video visualization can be found at https://www.ekkasit.com/ControlMM-page/

Related papers

• From Open Loop to Closed Loop: A Test-Time Iterative Optimization Framework for Reference-Consistent Image Generation [108.02425846269234]
  controllable image generation has made significant strides by incorporating visual reference conditions.<n>Existing methods predominantly operate as open-loop systems.<n>We propose a novel test-time iterative optimization framework that reformulates reference-consistent generation as a closed-loop dynamic tracking problem.
  arXiv  Detail & Related papers  (2026-07-06T05:36:30Z)
• Multi-scale Coarse-to-fine Modeling for Test-time Human Motion Control [51.92884966472683]
  MSCoT is a multi-scale, coarse-to-fine model for test-time human motion synthesis and control.<n> MSCoT discretizes motion into a multi-scale hierarchical representation and predicts the entire token sequence at each temporal scale.
  arXiv  Detail & Related papers  (2026-05-14T15:09:49Z)
• Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer [55.9892973179428]
  MoTok is a discrete motion tokenizer that decouples semantic abstraction from fine-grained reconstruction.<n>On HumanML3D, our method significantly improves controllability and fidelity over MaskControl while using only one-sixth of the tokens.
  arXiv  Detail & Related papers  (2026-03-19T17:59:51Z)
• Warm-Starting Collision-Free Model Predictive Control With Object-Centric Diffusion [38.363402738466334]
  We show that combining diffusion-based warm-starting conditioned with a latent object-centric representation of the scene yields reliable and efficient motion generation under strict time limits.<n>Our approach conditions a diffusion transformer on the system state, task, and surroundings, using an object-centric slot attention mechanism.<n>The sampled trajectories are refined by an optimal control problem that enforces rigid-body dynamics and signed-distance collision constraints.
  arXiv  Detail & Related papers  (2026-01-06T10:02:54Z)
• $\mathcal{E}_0$: Enhancing Generalization and Fine-Grained Control in VLA Models via Continuized Discrete Diffusion [65.77755100137728]
  We introduce E0, a continuized discrete diffusion framework that formulates action generation as iterative denoising over quantized action tokens.<n>E0 achieves state-of-the-art performance across 14 diverse environments, outperforming strong baselines by 10.7% on average.
  arXiv  Detail & Related papers  (2025-11-26T16:14:20Z)
• ScaleWeaver: Weaving Efficient Controllable T2I Generation with Multi-Scale Reference Attention [86.93601565563954]
  ScaleWeaver is a framework designed to achieve high-fidelity, controllable generation upon advanced visual autoregressive( VAR) models.<n>The proposed Reference Attention module discards the unnecessary attention from image$rightarrow$condition, reducing computational cost.<n>Experiments show that ScaleWeaver delivers high-quality generation and precise control while attaining superior efficiency over diffusion-based methods.
  arXiv  Detail & Related papers  (2025-10-16T17:00:59Z)
• RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS [79.15416002879239]
  3D Gaussian Splatting has gained significant attention for its real-time, photo-realistic rendering in novel-view synthesis and 3D modeling.<n>Existing methods struggle with accurately modeling scenes affected by transient objects, leading to artifacts in the rendered images.<n>We propose RobustSplat, a robust solution based on two critical designs.
  arXiv  Detail & Related papers  (2025-06-03T11:13:48Z)
• Interactive Video Generation via Domain Adaptation [7.397099215417549]
  Text-conditioned diffusion models have emerged as powerful tools for high-quality video generation.<n>Recent training-free approaches introduce attention masking to guide trajectory, but this often degrades quality.<n>We identify two key modes failure in these methods, both of which we interpret as domain problems.
  arXiv  Detail & Related papers  (2025-05-30T06:19:47Z)
• Towards Robust and Controllable Text-to-Motion via Masked Autoregressive Diffusion [33.9786226622757]
  We propose a robust motion generation framework MoMADiff to generate 3D human motion from text descriptions.<n>Our model supports flexible user-provided specification, enabling precise control over both spatial and temporal aspects of motion synthesis.<n>Our method consistently outperforms state-of-the-art models in motion quality, instruction fidelity, and adherence.
  arXiv  Detail & Related papers  (2025-05-16T09:06:15Z)
• Enabling Versatile Controls for Video Diffusion Models [18.131652071161266]
  VCtrl is a novel framework designed to enable fine control over pre-trained video diffusion models. Comprehensive experiments and human evaluations demonstrate VCtrl effectively enhances controllability and generation quality.
  arXiv  Detail & Related papers  (2025-03-21T09:48:00Z)
• Mojito: Motion Trajectory and Intensity Control for Video Generation [79.85687620761186]
  This paper introduces Mojito, a diffusion model that incorporates both motion trajectory and intensity control for text-to-video generation. Experiments demonstrate Mojito's effectiveness in achieving precise trajectory and intensity control with high computational efficiency.
  arXiv  Detail & Related papers  (2024-12-12T05:26:43Z)
• Omegance: A Single Parameter for Various Granularities in Diffusion-Based Synthesis [55.00448838152145]
  We show that we only need a single parameter $omega$ to effectively control granularity in diffusion-based synthesis.<n>This simple approach does not require model retraining or architectural modifications and incurs negligible computational overhead.<n>The method demonstrates impressive performance across various image and video synthesis tasks and is adaptable to advanced diffusion models.
  arXiv  Detail & Related papers  (2024-11-26T08:23:16Z)
• ControlAR: Controllable Image Generation with Autoregressive Models [40.74890550081335]
  We introduce ControlAR, an efficient framework for integrating spatial controls into autoregressive image generation models.<n>ControlAR exploits the conditional decoding method to generate the next image token conditioned on the per-token fusion between control and image tokens.<n>Results indicate that ControlAR surpasses previous state-of-the-art controllable diffusion models.
  arXiv  Detail & Related papers  (2024-10-03T17:28:07Z)
• Uniformly Accelerated Motion Model for Inter Prediction [38.34487653360328]
  In natural videos, there are usually multiple moving objects with variable velocity, resulting in complex motion fields that are difficult to represent compactly. In Versatile Video Coding (VVC), existing inter prediction methods assume uniform speed motion between consecutive frames. We introduce a uniformly accelerated motion model (UAMM) to exploit motion-related elements (velocity, acceleration) of moving objects between the video frames.
  arXiv  Detail & Related papers  (2024-07-16T09:46:29Z)
• MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model [29.93359157128045]
  This work introduces MotionLCM, extending controllable motion generation to a real-time level. We first propose the motion latent consistency model (MotionLCM) for motion generation, building upon the latent diffusion model. By adopting one-step (or few-step) inference, we further improve the runtime efficiency of the motion latent diffusion model for motion generation.
  arXiv  Detail & Related papers  (2024-04-30T17:59:47Z)
• ECNet: Effective Controllable Text-to-Image Diffusion Models [31.21525123716149]
  We introduce two innovative solutions for conditional text-to-image models. Firstly, we propose a Spatial Guidance (SGI) which enhances conditional detail by encoding text inputs with precise annotation information. Secondly, to overcome the issue of limited conditional supervision, we introduce Diffusion Consistency Loss. This encourages consistency between the latent code at each time step and the input signal, thereby enhancing the robustness and accuracy of the output.
  arXiv  Detail & Related papers  (2024-03-27T10:09:38Z)
• When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability [93.15085958220024]
  ControlNet excels at creating content that closely matches precise contours in user-provided masks. When these masks contain noise, as a frequent occurrence with non-expert users, the output would include unwanted artifacts. This paper first highlights the crucial role of controlling the impact of these inexplicit masks with diverse deterioration levels through in-depth analysis. An advanced Shape-aware ControlNet consisting of a deterioration estimator and a shape-prior modulation block is devised.
  arXiv  Detail & Related papers  (2024-03-01T11:45:29Z)
• MMM: Generative Masked Motion Model [10.215003912084944]
  MMM is a novel yet simple motion generation paradigm based on Masked Motion Model. By attending to motion and text tokens in all directions, MMM captures inherent dependency among motion tokens and semantic mapping between motion and text tokens. MMM is two orders of magnitude faster on a single mid-range GPU than editable motion diffusion models.
  arXiv  Detail & Related papers  (2023-12-06T16:35:59Z)
• Fine-grained Controllable Video Generation via Object Appearance and Context [74.23066823064575]
  We propose fine-grained controllable video generation (FACTOR) to achieve detailed control. FACTOR aims to control objects' appearances and context, including their location and category. Our method achieves controllability of object appearances without finetuning, which reduces the per-subject optimization efforts for the users.
  arXiv  Detail & Related papers  (2023-12-05T17:47:33Z)
• EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generation [57.539634387672656]
  Current state-of-the-art generative diffusion models have produced impressive results but struggle to achieve fast generation without sacrificing quality. We introduce Efficient Motion Diffusion Model (EMDM) for fast and high-quality human motion generation.
  arXiv  Detail & Related papers  (2023-12-04T18:58:38Z)
• OmniControl: Control Any Joint at Any Time for Human Motion Generation [46.293854851116215]
  We present a novel approach named OmniControl for incorporating flexible spatial control signals into a text-conditioned human motion generation model. We propose analytic spatial guidance that ensures the generated motion can tightly conform to the input control signals. At the same time, realism guidance is introduced to refine all the joints to generate more coherent motion.
  arXiv  Detail & Related papers  (2023-10-12T17:59:38Z)
• MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators [108.67006263044772]
  This paper presents a Motion General-Purpose generaTor (MotionGPT) that can use multimodal control signals. We first quantize multimodal control signals into discrete codes and then formulate them in a unified prompt instruction. Our MotionGPT demonstrates a unified human motion generation model with multimodal control signals by tuning a mere 0.4% of LLM parameters.
  arXiv  Detail & Related papers  (2023-06-19T12:58:17Z)
• Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation [79.8881514424969]
  Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery. We propose Cocktail, a pipeline to mix various modalities into one embedding.
  arXiv  Detail & Related papers  (2023-06-01T17:55:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.