Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation

• URL: http://arxiv.org/abs/2608.21425v1
• Date: Sun, 16 Aug 2026 07:50:30 GMT
• Title: Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation
• Abstract summary: Video generation is central to AI-powered content creation.<n>Despite progress in visual quality, three key challenges remain.<n>First, the reliability of reward signals is constrained by the quality of human preference data.<n>Second, standard scalar reward models collapse multi-aspect human preferences into a single value.<n>Third, in policy optimization, the widely adopted KL divergence imposes primarily local constraints.
• Score: 8.918851532959815
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video generation is central to AI-powered content creation. Aligning generated videos with human preferences is a key criterion for evaluating generation quality. Despite significant progress in visual quality, three key challenges remain. First, the reliability of reward signals is constrained by the quality of human preference data, which is often affected by subjective noise and bias. Second, standard scalar reward models collapse multi-aspect human preferences into a single value, leading to the loss of dynamic trade-offs across multiple preference dimensions. Third, in policy optimization, the widely adopted KL divergence imposes primarily local constraints and may fail to capture the global structure of human preferences. To address these challenges, we propose a unified preference-aware learning framework for video generation. First, we introduce elite-guided filtering to calibrate preference data and construct reliable supervision for reward model training. We then model video quality as a multidimensional reward distribution to capture the uncertainty inherent in human preferences, and use the Wasserstein distance to align the learned reward distribution with the empirical human preference distribution. Finally, we introduce Wasserstein-based distributional alignment into GRPO, guiding policy optimization to better match the global structure of human preferences over videos. Experiments on reward modeling and video generation demonstrate that our approach improves the reliability of reward signals and the perceptual consistency of generated videos. Our code is available at https://github.com/alignhs26/ahs.

Related papers

• VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation [79.34043102155964]
  We propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation.<n>We show VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations.
  arXiv  Detail & Related papers  (2026-08-19T06:53:14Z)
• Mind the Generative Details: Direct Localized Detail Preference Optimization for Video Diffusion Models [65.16788152626499]
  LocalDPO builds a novel framework for aligning video diffusion models with human preferences.<n>We show that LocalDPO consistently improves video fidelity, temporal coherence and human preference scores over other post-training approaches.
  arXiv  Detail & Related papers  (2026-01-07T16:32:17Z)
• Robust Preference Alignment via Directional Neighborhood Consensus [13.313830197011983]
  We introduce Robust Preference Selection (RPS), a post-hoc, training-free method by leveraging directional neighborhood consensus.<n>RPS samples multiple responses from a local neighborhood of related preferences to create a superior candidate pool.<n>Our work presents a practical, theoretically-grounded solution for enhancing the reliability of preference-aligned models.
  arXiv  Detail & Related papers  (2025-10-23T12:39:20Z)
• From Preferences to Prejudice: The Role of Alignment Tuning in Shaping Social Bias in Video Diffusion Models [69.4332879415364]
  We introduce VideoBiasEval, a framework for evaluating social representation in video generation.<n>VideoBiasEval employs an event-based prompting strategy to disentangle semantic content from actor attributes.<n>We conduct the first end-to-end analysis connecting biases in human preference datasets, their amplification in reward models, and their propagation through alignment-tuned video diffusion models.
  arXiv  Detail & Related papers  (2025-10-20T07:37:43Z)
• Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learning [13.0209477024596]
  Identity-GRPO is a human feedback-driven optimization pipeline for multi-human identity-preserving video generation.<n>We employ a GRPO variant tailored for multi-human consistency, which greatly enhances VACE and Phantom.<n> Experiments show that Identity-GRPO achieves up to 18.9% improvement in human consistency metrics over baseline methods.
  arXiv  Detail & Related papers  (2025-10-16T03:13:56Z)
• Multi-Metric Preference Alignment for Generative Speech Restoration [15.696247605348383]
  We propose a multi-metric preference alignment strategy for generative models.<n>We observe consistent and significant performance gains across three diverse generative paradigms.<n>Our aligned models can serve as powerful ''data annotators'', generating high-quality pseudo-labels.
  arXiv  Detail & Related papers  (2025-08-24T07:05:10Z)
• Generative Video Matting [57.186684844156595]
  Video matting has traditionally been limited by the lack of high-quality ground-truth data.<n>Most existing video matting datasets provide only human-annotated imperfect alpha and foreground annotations.<n>We introduce a novel video matting approach that can effectively leverage the rich priors from pre-trained video diffusion models.
  arXiv  Detail & Related papers  (2025-08-11T12:18:55Z)
• Improving Video Generation with Human Feedback [105.81833319891537]
  We develop a systematic pipeline that harnesses human feedback to mitigate video generation problems.<n>We introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy.
  arXiv  Detail & Related papers  (2025-01-23T18:55:41Z)
• MaxMin-RLHF: Alignment with Diverse Human Preferences [101.57443597426374]
  Reinforcement Learning from Human Feedback (RLHF) aligns language models to human preferences by employing a singular reward model derived from preference data.<n>We learn a mixture of preference distributions via an expectation-maximization algorithm to better represent diverse human preferences.<n>Our algorithm achieves an average improvement of more than 16% in win-rates over conventional RLHF algorithms.
  arXiv  Detail & Related papers  (2024-02-14T03:56:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.