MotionRFT: Unified Reinforcement Fine-Tuning for Text-to-Motion Generation

• URL: http://arxiv.org/abs/2603.27185v1
• Date: Sat, 28 Mar 2026 08:10:53 GMT
• Title: MotionRFT: Unified Reinforcement Fine-Tuning for Text-to-Motion Generation
• Abstract summary: We present a reinforcement fine-tuning framework for text-to-motion generative models.<n> MotionReward maps heterogeneous motions into a shared semantic space anchored by text.<n>Self-refinement Preference Learning enhances semantics without additional annotations.<n>Experiments validate the effectiveness of our framework, achieving FID 0.132 at 22.10 GB peak memory for MLD model and saving up to 15.22 GB over DRaFT.
• Score: 36.11192536249353
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Text-to-motion generation has advanced with diffusion- and flow-based generative models, yet supervised pretraining remains insufficient to align models with high-level objectives such as semantic consistency, realism, and human preference. Existing post-training methods have key limitations: they (1) target a specific motion representation, such as joints, (2) optimize a particular aspect, such as text-motion alignment, and may compromise other factors; and (3) incur substantial computational overhead, data dependence, and coarse-grained optimization. We present a reinforcement fine-tuning framework that comprises a heterogeneous-representation, multi-dimensional reward model, MotionReward, and an efficient, fine-grained fine-tuning method, EasyTune. To obtain a unified semantics representation, MotionReward maps heterogeneous motions into a shared semantic space anchored by text, enabling multidimensional reward learning; Self-refinement Preference Learning further enhances semantics without additional annotations. For efficient and effective fine-tuning, we identify the recursive gradient dependence across denoising steps as the key bottleneck, and propose EasyTune, which optimizes step-wise rather than over the full trajectory, yielding dense, fine-grained, and memory-efficient updates. Extensive experiments validate the effectiveness of our framework, achieving FID 0.132 at 22.10 GB peak memory for MLD model and saving up to 15.22 GB over DRaFT. It reduces FID by 22.9% on joint-based ACMDM, and achieves a 12.6% R-Precision gain and 23.3% FID improvement on rotation-based HY Motion. Our project page with code is publicly available.

Related papers

• Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation [14.492779733437082]
  Referring Remote Sensing Image seeks to localize and segment the target object or region specified by a natural language expression in a remote sensing image.<n>Existing RRSIS models have benefited from large-scale foundation models, but they predominantly rely on full fine-tuning.<n>We propose a novel framework, Semantic-driven Scale and Spatial Selection for Efficient Cross-modal Alignment.
  arXiv  Detail & Related papers  (2026-06-29T12:54:10Z)
• Harnessing Lightweight Transformer with Contextual Synergic Enhancement for Efficient 3D Medical Image Segmentation [59.711744386226194]
  We propose Light-UNETR, a lightweight transformer designed to achieve model efficiency.<n>Light-UNETR features a Lightweight Dimension Reductive Attention (LIDR) module, which reduces spatial and channel dimensions.<n>We also introduce a Contextual Synergic Enhancement (CSE) learning strategy, which aims to boost the data efficiency of Transformers.
  arXiv  Detail & Related papers  (2026-03-24T16:24:19Z)
• Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference [69.34278282513593]
  Preference Score Distillation (PSD) is an optimization-based framework for human-aligned text-to-3D synthesis without 3D training data.<n>Our key insight stems from the incompatibility of pixel-level gradients.<n>We introduce an adaptive strategy to co-optimize preference scores and negative text embeddings.
  arXiv  Detail & Related papers  (2026-03-02T08:23:36Z)
• ReMoT: Reinforcement Learning with Motion Contrast Triplets [37.29312323908102]
  We present ReMoT, a unified training paradigm to address the fundamental shortcomings of VLMs in-temporal.<n>A rule-based automatic framework generates ReMoT-16K triplets, a large-scale (1K) motion-contrast triplets from video meta-annotations.<n>We also construct the first benchmark for fine-grained motion contrast triplets to measure a VLMs of subtle discrimination.
  arXiv  Detail & Related papers  (2026-02-28T04:42:34Z)
• EasyTune: Efficient Step-Aware Fine-Tuning for Diffusion-Based Motion Generation [6.674252840296312]
  We propose EasyTune, which fine-tunes diffusion at each denoising step rather than over the entire trajectory.<n>We also introduce a Self-refinement Preference Learning mechanism that dynamically identifies preference pairs and conducts preference learning.<n>Experiments demonstrate that EasyTune outperforms DRaFT-50 by 8.2% in alignment (MM-Dist) improvement.
  arXiv  Detail & Related papers  (2026-02-08T13:29:46Z)
• Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations [55.047454145941366]
  Streaming Merging is an innovative model updating paradigm that conceptualizes merging as an iterative optimization process.<n> ARM is a strategy designed to approximate gradient descent dynamics.<n> ARM requires only early SFT checkpoints and, through iterative merging, surpasses the fully converged SFT model.
  arXiv  Detail & Related papers  (2026-02-03T08:15:57Z)
• ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer [58.49950218437718]
  We present ReCoM, an efficient framework for generating high-fidelity and generalizable human body motions synchronized with speech.<n>The core innovation lies in the Recurrent Embedded Transformer (RET), which integrates Dynamic Embedding Regularization (DER) into a Vision Transformer (ViT) core architecture.<n>To enhance model robustness, we incorporate the proposed DER strategy, which equips the model with dual capabilities of noise resistance and cross-domain generalization.
  arXiv  Detail & Related papers  (2025-03-27T16:39:40Z)
• Towards Generalizable Trajectory Prediction Using Dual-Level Representation Learning And Adaptive Prompting [107.4034346788744]
  Existing vehicle trajectory prediction models struggle with generalizability, prediction uncertainties, and handling complex interactions.<n>We propose Perceiver with Register queries (PerReg+), a novel trajectory prediction framework that introduces: (1) Dual-Level Representation Learning via Self-Distillation (SD) and Masked Reconstruction (MR), capturing global context and fine-grained details; (2) Enhanced Multimodality using register-based queries and pretraining, eliminating the need for clustering and suppression; and (3) Adaptive Prompt Tuning during fine-tuning, freezing the main architecture and optimizing a small number of prompts for efficient adaptation.
  arXiv  Detail & Related papers  (2025-01-08T20:11:09Z)
• ALOcc: Adaptive Lifting-Based 3D Semantic Occupancy and Cost Volume-Based Flow Predictions [91.55655961014027]
  3D semantic occupancy and flow prediction are fundamental to understanding scene scene.<n>This paper proposes a vision-based framework with three targeted improvements.<n>Our purely convolutional architecture establishes new SOTA performance on multiple benchmarks for both semantic occupancy and joint semantic-flow prediction.
  arXiv  Detail & Related papers  (2024-11-12T11:32:56Z)
• PoseRAC: Pose Saliency Transformer for Repetitive Action Counting [56.34379680390869]
  We introduce Pose Saliency Representation, which efficiently represents each action using only two salient poses instead of redundant frames. We also introduce PoseRAC, which is based on this representation and achieves state-of-the-art performance. Our lightweight model is highly efficient, requiring only 20 minutes for training on a GPU, and infers nearly 10x faster compared to previous methods.
  arXiv  Detail & Related papers  (2023-03-15T08:51:17Z)
• DWRSeg: Rethinking Efficient Acquisition of Multi-scale Contextual Information for Real-time Semantic Segmentation [10.379708894083217]
  We propose a highly efficient multi-scale feature extraction method, which decomposes the original single-step method into two steps, Region Residualization-Semantic Residualization. We achieve an mIoU of 72.7% on the Cityscapes test set at a speed of 319.5 FPS on one NVIDIA GeForce GTX 1080 Ti card, which exceeds the latest methods of a speed of 69.5 FPS and 0.8% mIoU.
  arXiv  Detail & Related papers  (2022-12-02T13:55:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.