ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control

• URL: http://arxiv.org/abs/2604.20816v1
• Date: Wed, 22 Apr 2026 17:44:56 GMT
• Title: ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control
• Abstract summary: Reinforcement Learning (RL) post-training has become the standard for aligning generative models with human preferences.<n>The prevailing practice of early scalarization'' collapses rewards into a fixed weighted sum when multiple criteria matter.<n>This commits the model to a single trade-off point at training time, providing no inference-time control over inherently conflicting goals.
• Score: 14.748861060405973
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning (RL) post-training has become the standard for aligning generative models with human preferences, yet most methods rely on a single scalar reward. When multiple criteria matter, the prevailing practice of ``early scalarization'' collapses rewards into a fixed weighted sum. This commits the model to a single trade-off point at training time, providing no inference-time control over inherently conflicting goals -- such as prompt adherence versus source fidelity in image editing. We introduce ParetoSlider, a multi-objective RL (MORL) framework that trains a single diffusion model to approximate the entire Pareto front. By training the model with continuously varying preference weights as a conditioning signal, we enable users to navigate optimal trade-offs at inference time without retraining or maintaining multiple checkpoints. We evaluate ParetoSlider across three state-of-the-art flow-matching backbones: SD3.5, FluxKontext, and LTX-2. Our single preference-conditioned model matches or exceeds the performance of baselines trained separately for fixed reward trade-offs, while uniquely providing fine-grained control over competing generative goals.

Related papers

• Test-Time Weak-to-Strong Alignment: Transferring Implicit Rewards from Weak to Strong Flow Models [19.4335297564654]
  Test-time alignment steers a frozen model during sampling, allowing task-specific and sample-specific guidance.<n>We propose changing the supervision source: let a pair of weak models, not a reward function, supply the supervision.<n>A source aligned model, kept together with its base as a source alignment pair, stores its training reward as an implicit, step-wise, KL-anchored signal.
  arXiv  Detail & Related papers  (2026-09-05T08:12:49Z)
• Controllable and explainable personality sliders for LLMs at inference time [1.3688381983244782]
  We propose a modular framework for continuous, multi-dimensional personality control.<n>Our key innovation is Sequential Adaptive Steering (SAS), a method that transforms steering vectors by training subsequent probes on the residual stream shifted by prior interventions.<n>We validate our framework on the Big Five personality traits, demonstrating that it outperforms naive baselines in both goal adherence and coherence.
  arXiv  Detail & Related papers  (2026-02-10T08:16:59Z)
• Improved Training Technique for Shortcut Models [12.527716901034694]
  Shortcut models are a promising, non-adversarial paradigm for generative modeling.<n>Shortcut models support one-step, few-step, and multi-step sampling from a single trained network.<n>This paper tackles the five core issues that held shortcut models back.
  arXiv  Detail & Related papers  (2025-10-24T08:35:04Z)
• Dual-Weighted Reinforcement Learning for Generative Preference Modeling [61.443461640955796]
  We propose Dual-Weighted Reinforcement Learning (DWRL) as a new framework for preference modeling.<n>In this paper, we apply DWRL to preference modeling by training generative preference models (GPMs) to first generate a thought and then predict the human preference score.<n>Our results position DWRL as a general framework for reasoning-enhanced preference learning beyond verifiable tasks.
  arXiv  Detail & Related papers  (2025-10-17T02:14:24Z)
• Optimal Control Meets Flow Matching: A Principled Route to Multi-Subject Fidelity [35.95129874095729]
  Text-to-image (T2I) models excel on single-entity prompts but struggle with multi-subject descriptions.<n>We introduce the first theoretical framework with principled optimizable objective for steering sampling dynamics toward multi-subject fidelity.
  arXiv  Detail & Related papers  (2025-10-02T17:59:58Z)
• Crowd-SFT: Crowdsourcing for LLM Alignment [4.648677931378919]
  Large Language Models (LLMs) increasingly rely on Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF)<n>We propose an open, crowd-sourced fine-tuning framework that enables broader feedback collection for SFT without extensive annotator training.<n>Our framework promotes incentive fairness via a point-based reward system correlated with Shapley values and guides model convergence through iterative model updates.
  arXiv  Detail & Related papers  (2025-06-04T15:26:38Z)
• Test-Time Alignment via Hypothesis Reweighting [56.71167047381817]
  Large pretrained models often struggle with underspecified tasks.<n>We propose a novel framework to address the challenge of aligning models to test-time user intent.
  arXiv  Detail & Related papers  (2024-12-11T23:02:26Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free [115.81899803240758]
  Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy. This paper asks how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies. Our proposed framework, Once-for-all Adversarial Training (OAT), is built on an innovative model-conditional training framework.
  arXiv  Detail & Related papers  (2020-10-22T16:06:34Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.