AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation

• URL: http://arxiv.org/abs/2506.11144v1
• Date: Wed, 11 Jun 2025 05:33:03 GMT
• Title: AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation
• Abstract summary: We propose textbfAlignHuman, a framework that combines Preference Optimization as a post-training technique with a divide-and-conquer training strategy.<n>LoRAs are trained using their respective preference data and activated in the corresponding intervals during inference to enhance motion naturalness and fidelity.<n>Experiments demonstrate that AlignHuman improves strong baselines and reduces NFEs during inference, achieving a 3.3$times$ speedup.
• Score: 24.745851101654612
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advancements in human video generation and animation tasks, driven by diffusion models, have achieved significant progress. However, expressive and realistic human animation remains challenging due to the trade-off between motion naturalness and visual fidelity. To address this, we propose \textbf{AlignHuman}, a framework that combines Preference Optimization as a post-training technique with a divide-and-conquer training strategy to jointly optimize these competing objectives. Our key insight stems from an analysis of the denoising process across timesteps: (1) early denoising timesteps primarily control motion dynamics, while (2) fidelity and human structure can be effectively managed by later timesteps, even if early steps are skipped. Building on this observation, we propose timestep-segment preference optimization (TPO) and introduce two specialized LoRAs as expert alignment modules, each targeting a specific dimension in its corresponding timestep interval. The LoRAs are trained using their respective preference data and activated in the corresponding intervals during inference to enhance motion naturalness and fidelity. Extensive experiments demonstrate that AlignHuman improves strong baselines and reduces NFEs during inference, achieving a 3.3$\times$ speedup (from 100 NFEs to 30 NFEs) with minimal impact on generation quality. Homepage: \href{https://alignhuman.github.io/}{https://alignhuman.github.io/}

Related papers

• StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models [84.17128030384099]
  StreamPI is a streaming multimodal temporal modeling framework.<n>It equips single-frame VLA with temporal reasoning capability without introducing any additional parameters.<n> Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
  arXiv  Detail & Related papers  (2026-08-26T17:33:19Z)
• BFMTrack: Latent Sequence Optimization for Physics-Based Motion Tracking with Behavioral Foundation Models [7.959692704349906]
  Behavioral Foundation Models (BFMs) organize a rich repertoire of physically plausible behaviors into a latent space, guided by a large-scale motion dataset.<n>These models excel at time-invariant tasks, such as goal-reaching and state-based reward optimization, but their latent space does not directly support time-varying objectives, such as tracking a motion sequence.<n>We propose a novel Latent Sequence Optimization (LSO) to address these shortcomings.
  arXiv  Detail & Related papers  (2026-06-23T18:12:22Z)
• Learning to Credit the Right Steps: Objective-aware Process Optimization for Visual Generation [57.61006679699323]
  Reinforcement learning has emerged as an effective framework for post-training visual generative models with human preference signals.<n>In modern visual generation, multiple reward models are often used to capture heterogeneous objectives, such as visual quality, motion consistency, and text alignment.<n>We propose Objective-aware Trajectory Credit Assignment (OTCA), a structured framework for fine-grained GRPO training.
  arXiv  Detail & Related papers  (2026-04-21T08:37:13Z)
• Unleashing Guidance Without Classifiers for Human-Object Interaction Animation [54.24065561632133]
  Prior diffusion-based approaches often rely on hand-crafted contact priors or human-imposed kinematic constraints to improve contact quality.<n>We propose a data-driven alternative in which guidance emerges from the denoising pace itself, reducing dependence on manually designed priors.<n>We find that this data-driven guidance is inherently contact-aware, and can be enhanced when training is augmented with a broad spectrum of synthetic object geometries.
  arXiv  Detail & Related papers  (2026-03-26T17:59:34Z)
• PEARL: Personalized Streaming Video Understanding Model [50.273809779498464]
  multimodal personalization methods are largely limited to static images or offline videos.<n>PEARL-Bench is the first comprehensive benchmark designed specifically to evaluate this challenging setting.<n>PEARL achieves state-of-the-art performance across 8 offline and online models.
  arXiv  Detail & Related papers  (2026-03-20T18:47:54Z)
• VAMPO: Policy Optimization for Improving Visual Dynamics in Video Action Models [46.94937828558026]
  We propose VAMPO, a post-training framework that directly improves visual dynamics in video action models through policy optimization.<n>Our key idea is to formulate multi-step denoising as a sequential decision process and optimize the denoising rewards defined over expert visual dynamics.<n>Across diverse simulated and real-world manipulation tasks, VAMPO improves task-relevant visual dynamics, leading to better downstream action generation and stronger generalization.
  arXiv  Detail & Related papers  (2026-03-19T18:04:03Z)
• JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation [112.614973927778]
  Joint audio-video generation (JAVG) produces synchronized and semantically aligned sound and vision from textual descriptions.<n>This paper presents JavisDiT++, a framework for unified modeling and optimization of JAVG.<n>Our model achieves state-of-the-art performance merely with around 1M public training entries.
  arXiv  Detail & Related papers  (2026-02-22T12:44:28Z)
• SARAH: Spatially Aware Real-time Agentic Humans [58.32612596034656]
  We present the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset.<n>Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user.<n>We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment.
  arXiv  Detail & Related papers  (2026-02-20T18:59:35Z)
• High-Fidelity and Long-Duration Human Image Animation with Diffusion Transformer [17.388852038062705]
  We propose a diffusion transformer (DiT)-based framework which focuses on generating high-fidelity and long-duration human animation videos.<n>First, we design a set of hybrid implicit guidance signals and a sharpness guidance factor, enabling our framework to additionally incorporate detailed facial and hand features as guidance.<n>Next, we incorporate the time-aware position shift fusion module, modify the input format within the DiT backbone, and refer to this mechanism as the Position Shift Adaptive Module.
  arXiv  Detail & Related papers  (2025-12-26T07:36:48Z)
• TalkingPose: Efficient Face and Gesture Animation with Feedback-guided Diffusion Model [18.910745982208965]
  TalkingPose is a novel diffusion-based framework for producing temporally consistent human upper-body animations.<n>We introduce a feedback-driven mechanism built upon image-based diffusion models to ensure continuous motion and enhance temporal coherence.<n>We also introduce a comprehensive, large-scale dataset to serve as a new benchmark for human upper-body animation.
  arXiv  Detail & Related papers  (2025-11-30T14:26:24Z)
• Planning with Sketch-Guided Verification for Physics-Aware Video Generation [71.29706409814324]
  We propose SketchVerify as a training-free, sketch-verification-based planning framework for video generation.<n>Our method predicts multiple candidate motion plans and ranks them using a vision-language verifier.<n>We iteratively refine the motion plan until a satisfactory one is identified, which is then passed to the trajectory-conditioned generator for final synthesis.
  arXiv  Detail & Related papers  (2025-11-21T17:48:02Z)
• Geometric Neural Distance Fields for Learning Human Motion Priors [51.99890740169883]
  We introduce a novel 3D generative human motion prior that enables robust, temporally consistent, and physically plausible 3D motion recovery.<n>Our experiments show significant and consistent gains: trained on the AMASS dataset, NRMF remarkably generalizes across multiple input modalities.
  arXiv  Detail & Related papers  (2025-09-11T17:58:18Z)
• M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation [65.08520614570288]
  We reformulate talking head generation into a unified framework comprising video preprocessing, motion representation, and rendering reconstruction.<n>M2DAO-Talker achieves state-of-the-art performance, with the 2.43 dB PSNR improvement in generation quality and 0.64 gain in user-evaluated video realness.
  arXiv  Detail & Related papers  (2025-07-11T04:48:12Z)
• GGMotion: Group Graph Dynamics-Kinematics Networks for Human Motion Prediction [9.723217255594793]
  GGMotion is a group graph dynamics-kinematics network that models human topology in groups to better leverage dynamics and kinematics priors.<n>Inter-group and intra-group interaction modules are employed to capture the dependencies of joints at different scales.<n>Our approach achieves a significant performance margin in short-term motion prediction.
  arXiv  Detail & Related papers  (2025-07-10T08:02:01Z)
• Zero-Shot Temporal Interaction Localization for Egocentric Videos [13.70694228506315]
  We propose a novel zero-shot TIL approach dubbed EgoLoc to locate the timings of grasp actions for human-object interaction in egocentric videos.<n>By absorbing both 2D and 3D observations, EgoLoc directly samples high-quality initial guesses around the possible contact/separation timestamps of HOI.<n>EgoLoc achieves better temporal interaction localization for egocentric videos compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-06-04T07:52:46Z)
• Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimization and Temporal Motion Modulation [26.597877504216196]
  We introduce direct preference optimization tailored for human-centric animation.<n>Second, the proposed temporal motion modulation resolves resolution mismatches.<n>Experiments demonstrate obvious improvements in lip-audio synchronization, expression vividness, body motion coherence over baseline methods.
  arXiv  Detail & Related papers  (2025-05-29T15:04:00Z)
• SuperFlow++: Enhanced Spatiotemporal Consistency for Cross-Modal Data Pretraining [62.433137130087445]
  SuperFlow++ is a novel framework that integrates pretraining and downstream tasks using consecutive camera pairs.<n>We show that SuperFlow++ outperforms state-of-the-art methods across diverse tasks and driving conditions.<n>With strong generalizability and computational efficiency, SuperFlow++ establishes a new benchmark for data-efficient LiDAR-based perception in autonomous driving.
  arXiv  Detail & Related papers  (2025-03-25T17:59:57Z)
• EvAnimate: Event-conditioned Image-to-Video Generation for Human Animation [58.41979933166173]
  EvAnimate is the first method leveraging event streams as robust and precise motion cues for conditional human image animation.<n>High-quality and temporally coherent animations are achieved through a dual-branch architecture.<n>Experiment results show EvAnimate achieves high temporal fidelity and robust performance in scenarios where traditional video-derived cues fall short.
  arXiv  Detail & Related papers  (2025-03-24T11:05:41Z)
• GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling [32.47567372398872]
  GestureLSM is a flow-matching-based approach for Co-Speech Gesture Generation with spatial-temporal modeling.<n>It achieves state-of-the-art performance on BEAT2 while significantly reducing inference time compared to existing methods.
  arXiv  Detail & Related papers  (2025-01-31T05:34:59Z)
• ALOcc: Adaptive Lifting-based 3D Semantic Occupancy and Cost Volume-based Flow Prediction [89.89610257714006]
  Existing methods prioritize higher accuracy to cater to the demands of these tasks. We introduce a series of targeted improvements for 3D semantic occupancy prediction and flow estimation. Our purelytemporalal architecture framework, named ALOcc, achieves an optimal tradeoff between speed and accuracy.
  arXiv  Detail & Related papers  (2024-11-12T11:32:56Z)
• MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward Reinforcement Learning [99.09906827676748]
  We introduce MotionRL, the first approach to utilize Multi-Reward Reinforcement Learning (RL) for optimizing text-to-motion generation tasks. Our novel approach uses reinforcement learning to fine-tune the motion generator based on human preferences prior knowledge of the human perception model. In addition, MotionRL introduces a novel multi-objective optimization strategy to approximate optimality between text adherence, motion quality, and human preferences.
  arXiv  Detail & Related papers  (2024-10-09T03:27:14Z)
• TIMotion: Temporal and Interactive Framework for Efficient Human-Human Motion Generation [30.734182958106327]
  Current methods fall into two main categories: single-person-based methods and separate modeling-based methods.<n>We introduce TIMotion (Temporal and Interactive Modeling), an efficient and effective framework for human-human motion generation.
  arXiv  Detail & Related papers  (2024-08-30T09:22:07Z)
• Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer [62.29951737214263]
  Existing algorithms directly generate the full sequence which is expensive and prone to errors. We propose KeyMotion, that generates plausible human motion sequences corresponding to input text. We use a Variationalcoder (VAE) with Kullback-Leibler regularization to project the Autoencoder into a latent space. For the reverse diffusion, we propose a novel Parallel Skip Transformer that performs cross-modal attention between the design latents and text condition.
  arXiv  Detail & Related papers  (2024-05-24T11:12:37Z)
• Motion-DVAE: Unsupervised learning for fast human motion denoising [18.432026846779372]
  We introduce Motion-DVAE, a motion prior to capture the short-term dependencies of human motion. Together with Motion-DVAE, we introduce an unsupervised learned denoising method unifying regression- and optimization-based approaches.
  arXiv  Detail & Related papers  (2023-06-09T12:18:48Z)
• GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation [71.73912454164834]
  A modern talking face generation method is expected to achieve the goals of generalized audio-lip synchronization, good video quality, and high system efficiency. NeRF has become a popular technique in this field since it could achieve high-fidelity and 3D-consistent talking face generation with a few-minute-long training video. We propose GeneFace++ to handle these challenges by utilizing the rendering pitch contour as an auxiliary feature and introducing a temporal loss in the facial motion prediction process.
  arXiv  Detail & Related papers  (2023-05-01T12:24:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.