When Preference Labels Fall Short: Aligning Diffusion Models from Real Data

• URL: http://arxiv.org/abs/2605.19839v1
• Date: Tue, 19 May 2026 13:35:11 GMT
• Title: When Preference Labels Fall Short: Aligning Diffusion Models from Real Data
• Abstract summary: Preference alignment aims to guide generative models by learning from comparisons between preferred and non-preferred samples.<n>Most existing approaches rely on preference pairs constructed from model-generated images.<n>We investigate whether real data can serve as an alternative source of supervision for preference alignment.
• Score: 65.80726604713614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Preference alignment aims to guide generative models by learning from comparisons between preferred and non-preferred samples. In practice, most existing approaches rely on preference pairs constructed from model-generated images. Such supervision is inherently relative and can be ambiguous when both samples exhibit artifacts or limited visual quality, making it difficult to infer what constitutes a truly desirable output. In this work, we investigate whether real data can serve as an alternative source of supervision for preference alignment. We adopt a data-centric perspective and study a curation strategy that treats real images as reference points and constructs preference signals by contrasting them with generated or perturbed samples, without requiring manually annotated preference pairs. Through empirical analysis, we show that real-data-based supervision provides effective guidance for aligning diffusion models and achieves performance comparable to existing preference-based methods. Our results suggest that real data offers a practical and complementary source of supervision for preference alignment and highlight directions of label-efficient alignment strategies. Code and models are available at https://cwyxx.github.io/RealAlign.

Related papers

• AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection [53.61691036434302]
  We propose AlignDiff, a preference data filtering framework driven by intrinsic model signals.<n>AlpacaEval 2.0, Arena-Hard, and MT-Bench benchmarks are evaluated.<n>We conduct comprehensive ablation studies to validate the effectiveness of AlignDiff.
  arXiv  Detail & Related papers  (2026-09-05T05:51:23Z)
• Threshold-Guided Optimization for Visual Generative Models [16.556017381410943]
  We propose a threshold-guided framework for aligning visual generative models with human feedback.<n>Our method consistently improves preference alignment over previous methods.<n>These results position our threshold-guided framework as a simple yet principled alternative for aligning visual generative models without paired comparisons.
  arXiv  Detail & Related papers  (2026-05-06T08:59:16Z)
• How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics [65.67654005892469]
  We show that proper instance-dependent sampling can yield stronger ranking guarantees, while skewed on-policy sampling can induce excessive concentration under structured preferences.<n>We then analyze iterative alignment dynamics in which the learned policy feeds back into future sampling and reference policies.<n>Our theoretical insights extend to Direct Preference Optimization, indicating the phenomena we captured are common to a broader class of preference-alignment methods.
  arXiv  Detail & Related papers  (2026-02-12T17:11:08Z)
• Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision [14.612317970237436]
  Diffusion models have achieved impressive results in generative tasks such as text-to-image synthesis.<n>They often struggle to fully align outputs with nuanced user intent and maintain consistent aesthetic quality.<n>Existing preference-based training methods help address these issues but rely on costly and potentially noisy human datasets.
  arXiv  Detail & Related papers  (2025-12-29T12:46:07Z)
• Nonparametric Data Attribution for Diffusion Models [57.820618036556084]
  Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs.<n>We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images.
  arXiv  Detail & Related papers  (2025-10-16T03:37:16Z)
• Towards Understanding Valuable Preference Data for Large Language Model Alignment [85.38864561060088]
  Large language model (LLM) alignment is typically achieved through learning from human preference comparisons.<n>We assess data quality through individual influence on validation data using our newly proposed truncated influence function (TIF)<n>To this end, we combine them to offset their diverse error sources, resulting in a simple yet effective data selection rule.
  arXiv  Detail & Related papers  (2025-10-15T06:57:55Z)
• Follow-Your-Preference: Towards Preference-Aligned Image Inpainting [17.648992293002088]
  We leverage the prominent direct preference optimization approach for alignment training.<n>We employ public reward models to construct preference training datasets.<n>Our work can set a simple yet solid baseline, pushing this promising frontier.
  arXiv  Detail & Related papers  (2025-09-27T03:32:30Z)
• Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization [67.8738082040299]
  Self-Sampling Preference Optimization (SSPO) is a new alignment method for post-training reinforcement learning.<n>SSPO eliminates the need for paired data and reward models while retaining the training stability of SFT.<n>SSPO surpasses all previous approaches on the text-to-image benchmarks and demonstrates outstanding performance on the text-to-video benchmarks.
  arXiv  Detail & Related papers  (2024-10-07T17:56:53Z)
• Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient [52.2669490431145]
  PropEn is inspired by'matching', which enables implicit guidance without training a discriminator. We show that training with a matched dataset approximates the gradient of the property of interest while remaining within the data distribution.
  arXiv  Detail & Related papers  (2024-05-28T11:30:19Z)
• ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference [16.73260713938154]
  A typical alignment procedure consists of supervised fine-tuning and preference learning. We introduce Point-wise Direct Preference Optimization, a novel preference learning method designed to harness point-wise feedback effectively. Our work also uncovers a novel connection between supervised fine-tuning and point-wise preference learning, culminating in Unified Language Model Alignment.
  arXiv  Detail & Related papers  (2023-12-05T07:52:12Z)
• Graph-Based Model-Agnostic Data Subsampling for Recommendation Systems [29.713557081485995]
  Data subsampling is widely used to speed up the training of recommendation systems. Most subsampling methods are model-based and often require a pre-trained pilot model to measure data importance. We propose model-agnostic data subsampling methods by only exploring input data structure represented by graphs.
  arXiv  Detail & Related papers  (2023-05-25T18:00:15Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.